Nov 03 00:04:37.979985 ip-10-0-135-199 systemd[1]: Starting Kubernetes Kubelet... Nov 03 00:04:38.319658 ip-10-0-135-199 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:04:38.319658 ip-10-0-135-199 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 00:04:38.319658 ip-10-0-135-199 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:04:38.320288 ip-10-0-135-199 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:04:38.324191 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324112 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:38.324191 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324191 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324195 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324199 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324203 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324205 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324209 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324212 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324219 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324221 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324224 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324227 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324229 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324232 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324234 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324237 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324239 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324242 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324244 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324247 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324251 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:38.324253 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324255 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324258 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324262 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324265 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324268 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324271 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324273 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324276 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324279 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324281 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324284 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324286 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324289 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324291 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324294 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324296 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324298 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324301 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324304 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:38.324710 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324307 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324315 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324317 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324320 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324322 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324325 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324327 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324329 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324332 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324334 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324337 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324339 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324342 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324344 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324347 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324350 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324352 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324355 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324357 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324360 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:38.325150 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324362 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324365 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324367 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324370 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324372 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324375 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324377 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324379 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324382 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324384 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324386 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324389 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324392 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324394 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324397 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324399 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324401 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324404 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324406 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:38.325601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324408 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324411 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324413 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324415 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324418 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324420 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324422 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324425 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324428 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324431 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324434 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324436 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324438 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324441 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324443 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324445 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324448 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324450 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324452 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:38.326062 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324454 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324457 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324459 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324462 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324464 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324466 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324469 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.324473 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325256 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325263 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325266 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325268 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325271 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325274 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325276 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325279 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325281 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325284 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325287 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:38.326623 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325289 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325293 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325296 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325299 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325302 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325304 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325307 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325311 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325315 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325318 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325321 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325324 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325327 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325329 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325331 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325334 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325337 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325339 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325341 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:38.327085 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325344 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325346 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325349 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325352 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325354 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325357 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325359 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325362 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325364 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325367 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325369 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325371 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325374 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325377 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325381 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325384 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325387 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325389 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325392 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325394 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:38.327532 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325396 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325399 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325401 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325404 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325406 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325409 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325411 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325413 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325416 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325418 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325420 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325423 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325425 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325428 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325430 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325434 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325437 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325439 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325442 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:38.328003 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325445 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325447 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325449 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325452 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325454 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325457 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325459 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325462 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325464 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325467 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325469 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325472 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325474 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325476 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325479 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325481 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325483 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325486 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325488 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:38.328437 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325506 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325510 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325515 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325518 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325521 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325524 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325526 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325529 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325532 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325534 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325537 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325548 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325551 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325554 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325557 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325560 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325562 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.325565 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326056 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:38.328899 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326061 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326064 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326067 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326070 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326073 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326076 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326078 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326081 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326083 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326086 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326088 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326091 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326093 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326096 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326098 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326101 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326103 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326105 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326108 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326111 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:38.329367 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326114 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326116 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326119 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326122 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326124 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326127 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326129 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326132 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326134 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326136 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326138 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326141 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326143 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326146 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326149 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326151 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326153 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326156 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326158 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326160 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:38.329833 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326163 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326165 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326168 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326170 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326173 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326175 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326178 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326180 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326183 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326185 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326187 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326190 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326193 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326195 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326197 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326200 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326203 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326205 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326208 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326211 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:38.330299 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326213 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326216 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326218 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326220 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326223 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326226 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326229 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326232 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326234 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326236 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326238 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326241 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326243 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326246 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326248 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326251 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326253 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326256 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326259 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326263 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:38.330818 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326266 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326270 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326274 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326278 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326281 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326283 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326286 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326288 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326290 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326294 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326296 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326299 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326302 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326304 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326306 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326309 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326311 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326313 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326316 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:38.331288 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326318 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326321 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326323 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326325 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326327 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.326330 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327080 2571 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327094 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327102 2571 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327107 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327111 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327117 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327121 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327125 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327128 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327131 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327134 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327138 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327141 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327144 2571 flags.go:64] FLAG: --cgroup-root="" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327146 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327149 2571 flags.go:64] FLAG: --client-ca-file="" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327152 2571 flags.go:64] FLAG: --cloud-provider="external" Nov 03 00:04:38.331808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327156 2571 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327161 2571 flags.go:64] FLAG: --cluster-domain="" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327163 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327166 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327169 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327172 2571 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327176 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327179 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327182 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327186 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327188 2571 flags.go:64] FLAG: --contention-profiling="false" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327191 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327194 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327197 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327200 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327204 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327207 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327210 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327213 2571 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327215 2571 flags.go:64] FLAG: --enable-server="true" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327220 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327225 2571 flags.go:64] FLAG: --event-burst="100" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327228 2571 flags.go:64] FLAG: --event-qps="50" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327231 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 00:04:38.332331 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327233 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327237 2571 flags.go:64] FLAG: --eviction-hard="" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327241 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327244 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327246 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327249 2571 flags.go:64] FLAG: --eviction-soft="" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327252 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327255 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327258 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327262 2571 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327264 2571 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327267 2571 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327270 2571 flags.go:64] FLAG: --feature-gates="" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327274 2571 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327277 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327280 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327283 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327286 2571 flags.go:64] FLAG: --healthz-port="10248" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327288 2571 flags.go:64] FLAG: --help="false" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327291 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327294 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327297 2571 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327300 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 00:04:38.332921 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327303 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327307 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327309 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327312 2571 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327315 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327319 2571 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327322 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327325 2571 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327328 2571 flags.go:64] FLAG: --kube-reserved="" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327331 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327333 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327336 2571 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327339 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327342 2571 flags.go:64] FLAG: --lock-file="" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327344 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327347 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327350 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327356 2571 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327358 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327362 2571 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327365 2571 flags.go:64] FLAG: --logging-format="text" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327367 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327371 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327374 2571 flags.go:64] FLAG: --manifest-url="" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327376 2571 flags.go:64] FLAG: --manifest-url-header="" Nov 03 00:04:38.333453 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327381 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327383 2571 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327387 2571 flags.go:64] FLAG: --max-pods="110" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327390 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327393 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327413 2571 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327416 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327420 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327423 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327426 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327432 2571 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327435 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327439 2571 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327442 2571 flags.go:64] FLAG: --pod-cidr="" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327446 2571 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327449 2571 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327452 2571 flags.go:64] FLAG: --pods-per-core="0" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327454 2571 flags.go:64] FLAG: --port="10250" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327458 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327460 2571 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0ffbcdb5cf358393a" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327463 2571 flags.go:64] FLAG: --qos-reserved="" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327466 2571 flags.go:64] FLAG: --read-only-port="10255" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327469 2571 flags.go:64] FLAG: --register-node="true" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327472 2571 flags.go:64] FLAG: --register-with-taints="" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327476 2571 flags.go:64] FLAG: --registry-burst="10" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327478 2571 flags.go:64] FLAG: --registry-qps="5" Nov 03 00:04:38.334089 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327481 2571 flags.go:64] FLAG: --reserved-cpus="" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327484 2571 flags.go:64] FLAG: --reserved-memory="" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327488 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327491 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327495 2571 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327497 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327500 2571 flags.go:64] FLAG: --runonce="false" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327503 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327506 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327509 2571 flags.go:64] FLAG: --seccomp-default="false" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327512 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327514 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327517 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327520 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327523 2571 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327526 2571 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327529 2571 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327531 2571 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327535 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327538 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327543 2571 flags.go:64] FLAG: --system-cgroups="" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327546 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327551 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327554 2571 flags.go:64] FLAG: --tls-cert-file="" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327557 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 00:04:38.334696 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327561 2571 flags.go:64] FLAG: --tls-min-version="" Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327564 2571 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327566 2571 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327569 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327572 2571 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327575 2571 flags.go:64] FLAG: --v="2" Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327579 2571 flags.go:64] FLAG: --version="false" Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327583 2571 flags.go:64] FLAG: --vmodule="" Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327587 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.327591 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327743 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327748 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327751 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327754 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327756 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327758 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327761 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327763 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327766 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327768 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327771 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327774 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:38.335293 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327776 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327778 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327781 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327784 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327787 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327790 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327793 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327796 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327798 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327800 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327803 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327805 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327808 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327810 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327813 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327815 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327818 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327820 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327823 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:38.335812 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327825 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327828 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327830 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327833 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327835 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327837 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327840 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327842 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327845 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327847 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327850 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327852 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327854 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327857 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327859 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327861 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327864 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327866 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327871 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327873 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:38.336247 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327876 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327878 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327882 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327885 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327889 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327892 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327895 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327897 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327899 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327902 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327904 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327907 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327909 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327911 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327914 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327917 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327919 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327921 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327924 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:38.336725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327926 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327928 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327931 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327933 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327936 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327938 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327940 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327943 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327945 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327948 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327950 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327954 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327957 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327959 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327962 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327964 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327966 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327969 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327971 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327974 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:38.337167 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327976 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327979 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327981 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327984 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327986 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327988 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327991 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327993 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327996 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.327999 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328002 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328004 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328007 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328009 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328013 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:38.337640 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328017 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.328045 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328085 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328089 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328092 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328095 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328099 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328102 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328107 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328109 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328112 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328115 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328118 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328120 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328123 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:38.338006 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328125 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328128 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328130 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328133 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328135 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328137 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328140 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328142 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328145 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328147 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328150 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328152 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328155 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328157 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328159 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328162 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328164 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328166 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328169 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:38.338354 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328171 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328173 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328176 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328178 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328181 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328184 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328188 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328190 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328193 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328195 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328198 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328201 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328203 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328206 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328209 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328212 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328214 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328217 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328219 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328221 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:38.338807 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328224 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328226 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328229 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328231 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328233 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328236 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328239 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328242 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328246 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328249 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328252 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328254 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328257 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328259 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328262 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328264 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328266 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328270 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328274 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:38.339259 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328276 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328278 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328281 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328283 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328286 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328288 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328290 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328292 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328295 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328298 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328300 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328302 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328305 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328307 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328310 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328312 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328314 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328317 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328319 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328322 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:38.339725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328324 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328327 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328329 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328331 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328334 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328336 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328338 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328341 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328343 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328345 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328349 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328354 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328356 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328359 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.328361 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.335206 2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.335220 2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.335778 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.335785 2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335843 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:38.340193 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335848 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335852 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335855 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335858 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335861 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335864 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335868 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335872 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335874 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335877 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335879 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335882 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335885 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335888 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335891 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335893 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335896 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335898 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335901 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335903 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:38.340716 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335905 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335908 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335911 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335914 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335918 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335923 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335926 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335929 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335932 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335935 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335938 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335940 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335943 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335945 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335948 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335951 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335953 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335955 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335957 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:38.341173 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335960 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335963 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335965 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335968 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335970 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335973 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335976 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335978 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335981 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335983 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335985 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335988 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335990 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335993 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335995 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.335998 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336000 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336003 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336005 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:38.341608 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336008 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336010 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336012 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336015 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336017 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336020 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336022 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336024 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336027 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336029 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336031 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336034 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336036 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336040 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336043 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336045 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336048 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336050 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336053 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:38.342054 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336055 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336058 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336060 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336063 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336065 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336067 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336070 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336072 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336075 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336078 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336080 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336083 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336085 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336088 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336090 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336092 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336095 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336097 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336100 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:38.342778 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336102 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336104 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336106 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336109 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336111 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336114 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336116 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336118 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336121 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.336148 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336277 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336282 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336285 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336287 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336290 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336293 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:38.343481 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336295 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336298 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336300 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336302 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336305 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336308 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336310 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336313 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336316 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336319 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336321 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336324 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336328 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336331 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336334 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336337 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336340 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336343 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336345 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:38.343915 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336348 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336350 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336353 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336355 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336358 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336360 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336362 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336365 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336367 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336369 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336372 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336385 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336388 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336391 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336393 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336396 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336398 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336401 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336404 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:38.344393 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336407 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336409 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336412 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336414 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336417 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336419 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336422 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336425 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336427 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336429 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336432 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336434 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336437 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336439 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336441 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336444 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336446 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336449 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336451 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336454 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:38.344847 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336456 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336459 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336461 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336463 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336466 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336468 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336471 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336474 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336477 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336479 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336481 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336484 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336487 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336489 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336492 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336494 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336497 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336499 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336502 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336504 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:38.345289 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336507 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336509 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336511 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336514 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336516 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336519 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336521 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336523 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336526 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336528 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336531 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336534 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336536 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336539 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336543 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336546 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336549 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336552 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336554 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:38.345787 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336557 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336560 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336562 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.336585 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336615 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336619 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336622 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336624 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336627 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336630 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336633 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336635 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336637 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336640 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336642 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336645 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:38.346226 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336647 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336650 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336652 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336655 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336658 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336660 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336662 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336678 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336682 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336686 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336688 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336691 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336694 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336696 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336698 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336701 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336704 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336709 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336712 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:38.346592 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336714 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336717 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336720 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336722 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336725 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336727 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336730 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336732 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336735 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336738 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336740 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336742 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336745 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336747 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336750 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336752 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336754 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336757 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336759 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:38.347043 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336761 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336764 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336767 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336769 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336773 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336777 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336779 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336782 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336785 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336788 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336790 2571 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336793 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336799 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336802 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336805 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336807 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336810 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336812 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336815 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:38.347511 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336817 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336819 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336823 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336825 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336827 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336830 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336832 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336835 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336837 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336839 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336845 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336848 2571 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336850 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336853 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336855 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336858 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336860 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336862 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336865 2571 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336868 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:38.347986 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336870 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336873 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336875 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336877 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336880 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336882 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336885 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336887 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336890 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336892 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336894 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336897 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336899 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336901 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336903 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336906 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:38.336908 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.336957 2571 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.339003 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 00:04:38.348436 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.339852 2571 server.go:1014] "Starting client certificate rotation" Nov 03 00:04:38.348909 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.339943 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:04:38.348909 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.339985 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:04:38.348909 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.346109 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:04:38.348909 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.347244 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:04:38.357865 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.357845 2571 log.go:25] "Validated CRI v1 runtime API" Nov 03 00:04:38.361130 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.361117 2571 log.go:25] "Validated CRI v1 image API" Nov 03 00:04:38.362047 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.362019 2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 00:04:38.367763 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.367745 2571 fs.go:135] Filesystem UUIDs: map[794c63d6-6a4f-47cf-895c-07c128ca5477:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e99209dc-730c-4191-929e-252c875033b6:/dev/nvme0n1p3] Nov 03 00:04:38.367826 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.367766 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 00:04:38.373352 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.373249 2571 manager.go:217] Machine: {Timestamp:2026-11-03 00:04:38.371934245 +0000 UTC m=+0.289689545 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3207207 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25ef4360106dd448bd3188962a8553 SystemUUID:ec25ef43-6010-6dd4-48bd-3188962a8553 BootID:46ad9331-ebf9-478e-9986-d46b099757f4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:03:56:a8:b1:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:03:56:a8:b1:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:d2:06:bb:ff:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 00:04:38.373429 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.373356 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 00:04:38.373474 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.373462 2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 00:04:38.375336 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.375301 2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 00:04:38.375467 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.375338 2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-199.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 00:04:38.375962 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.375953 2571 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 00:04:38.375990 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.375964 2571 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 00:04:38.376016 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.375990 2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:04:38.376016 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.376003 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:04:38.376641 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.376633 2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 00:04:38.380423 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.380411 2571 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 00:04:38.380550 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.380541 2571 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 00:04:38.381161 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.381153 2571 kubelet.go:493] "Attempting to sync node with API server" Nov 03 00:04:38.381189 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.381167 2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 00:04:38.381189 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.381177 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 00:04:38.381189 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.381185 2571 kubelet.go:405] "Adding apiserver pod source" Nov 03 00:04:38.381301 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.381195 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 00:04:38.381537 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.381521 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:04:38.383779 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.383758 2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 00:04:38.389221 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.389202 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:04:38.389926 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.389911 2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 00:04:38.389988 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.389941 2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 00:04:38.390109 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.390099 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 00:04:38.390139 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.390114 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 00:04:38.390139 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.390121 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 00:04:38.390139 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.390126 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 00:04:38.390139 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.390131 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 00:04:38.390139 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.390137 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 00:04:38.390629 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.390621 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 00:04:38.390680 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.390630 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 00:04:38.390680 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.390638 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 00:04:38.390680 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.390643 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 00:04:38.390680 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.390652 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 00:04:38.390680 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.390661 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 00:04:38.391219 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.391209 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 00:04:38.391219 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.391219 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 00:04:38.394881 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.394869 2571 server.go:1263] "Started kubelet" Nov 03 00:04:38.394961 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.394928 2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 00:04:38.395011 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.394959 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 00:04:38.395052 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.395034 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 00:04:38.395697 ip-10-0-135-199 systemd[1]: Started Kubernetes Kubelet. Nov 03 00:04:38.396466 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.396359 2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 00:04:38.397633 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.397621 2571 server.go:317] "Adding debug handlers to kubelet server" Nov 03 00:04:38.401365 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:38.401347 2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 00:04:38.403725 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.403702 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 00:04:38.404297 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.404283 2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 00:04:38.404466 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:38.404439 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-199.us-west-1.compute.internal\" not found" Nov 03 00:04:38.404887 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.404870 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 00:04:38.405013 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.404879 2571 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 00:04:38.405013 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.405009 2571 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 00:04:38.405145 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.405076 2571 reconstruct.go:97] "Volume reconstruction finished" Nov 03 00:04:38.405145 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.405085 2571 reconciler.go:29] "Reconciler: start to sync state" Nov 03 00:04:38.406619 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.406603 2571 factory.go:55] Registering systemd factory Nov 03 00:04:38.406619 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.406620 2571 factory.go:223] Registration of the systemd container factory successfully Nov 03 00:04:38.406867 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.406853 2571 factory.go:153] Registering CRI-O factory Nov 03 00:04:38.406944 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.406870 2571 factory.go:223] Registration of the crio container factory successfully Nov 03 00:04:38.406944 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.406920 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 00:04:38.406944 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.406937 2571 factory.go:103] Registering Raw factory Nov 03 00:04:38.407065 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.406951 2571 manager.go:1199] Started watching for new ooms in manager Nov 03 00:04:38.407976 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.407925 2571 manager.go:319] Starting recovery of all containers Nov 03 00:04:38.418444 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.418428 2571 manager.go:324] Recovery completed Nov 03 00:04:38.423347 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.423334 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:38.425087 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.425071 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:38.425142 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.425107 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:38.425142 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.425120 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:38.425631 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.425619 2571 cpu_manager.go:226] "Starting" policy="none" Nov 03 00:04:38.425631 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.425629 2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 00:04:38.425771 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.425633 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:38.425771 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.425645 2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 00:04:38.427222 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.427208 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:38.427300 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.427240 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:38.427300 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.427254 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:38.428047 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.428035 2571 policy_none.go:50] "Start" Nov 03 00:04:38.428047 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.428050 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 00:04:38.428143 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.428060 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 00:04:38.430164 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.430148 2571 policy_none.go:44] "Start" Nov 03 00:04:38.469984 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.469814 2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 00:04:38.513386 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.494625 2571 manager.go:341] "Starting Device Plugin manager" Nov 03 00:04:38.513386 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:38.494664 2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 00:04:38.513386 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.494693 2571 server.go:87] "Starting device plugin registration server" Nov 03 00:04:38.513386 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.494946 2571 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 00:04:38.513386 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.494957 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 00:04:38.513386 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.495097 2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 00:04:38.513386 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.495221 2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 00:04:38.513386 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.495230 2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 00:04:38.513386 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:38.495821 2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 00:04:38.513386 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:38.495889 2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-199.us-west-1.compute.internal\" not found" Nov 03 00:04:38.536840 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.536819 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-199.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 00:04:38.536951 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:38.536849 2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-199.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 00:04:38.537523 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.537496 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 00:04:38.538847 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.538832 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 00:04:38.538935 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.538856 2571 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 00:04:38.538935 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.538879 2571 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 00:04:38.538935 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:38.538915 2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 00:04:38.540139 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:38.536835 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-199.us-west-1.compute.internal.18e4602870082825 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-199.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-199.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 00:04:38.394841125 +0000 UTC m=+0.312596423,LastTimestamp:2026-11-03 00:04:38.394841125 +0000 UTC m=+0.312596423,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-199.us-west-1.compute.internal,}" Nov 03 00:04:38.595988 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.595970 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:38.596931 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.596916 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:38.596999 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.596944 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:38.596999 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.596955 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:38.596999 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.596981 2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.615181 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.615160 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 00:04:38.615248 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.615211 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5lc87" Nov 03 00:04:38.639207 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.639183 2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal"] Nov 03 00:04:38.639283 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.639251 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:38.640450 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.640435 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:38.640531 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.640463 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:38.640531 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.640473 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:38.642439 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.642426 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:38.642590 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.642576 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.642629 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.642608 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:38.643158 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.643141 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:38.643250 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.643171 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:38.643250 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.643180 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:38.643250 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.643141 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:38.643333 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.643259 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:38.643333 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.643271 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:38.644989 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.644972 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.645092 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.645002 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:38.645769 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.645743 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:38.645854 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.645775 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:38.645854 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.645789 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:38.661506 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:38.661489 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-199.us-west-1.compute.internal\" not found" node="ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.663613 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:38.663598 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-199.us-west-1.compute.internal\" not found" node="ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.693458 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.693440 2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:38.707367 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.707350 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bdf730cee51b9b033223ceb412560cad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"bdf730cee51b9b033223ceb412560cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.707465 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.707373 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdf730cee51b9b033223ceb412560cad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"bdf730cee51b9b033223ceb412560cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.707465 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.707398 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf58af1ed7bc599c71fbbaa4e8065ef6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"cf58af1ed7bc599c71fbbaa4e8065ef6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.791637 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.791609 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:38.791637 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.791626 2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:38.792393 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.792381 2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.792437 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:38.792399 2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-199.us-west-1.compute.internal\": node \"ip-10-0-135-199.us-west-1.compute.internal\" not found" Nov 03 00:04:38.792437 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.792405 2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:38.804851 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.804835 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.808068 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.808054 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bdf730cee51b9b033223ceb412560cad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"bdf730cee51b9b033223ceb412560cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.808127 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.808078 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdf730cee51b9b033223ceb412560cad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"bdf730cee51b9b033223ceb412560cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.808127 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.808095 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf58af1ed7bc599c71fbbaa4e8065ef6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"cf58af1ed7bc599c71fbbaa4e8065ef6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.808226 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.808142 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf58af1ed7bc599c71fbbaa4e8065ef6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"cf58af1ed7bc599c71fbbaa4e8065ef6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.808226 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.808174 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bdf730cee51b9b033223ceb412560cad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"bdf730cee51b9b033223ceb412560cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.808226 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.808176 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdf730cee51b9b033223ceb412560cad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"bdf730cee51b9b033223ceb412560cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.863587 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.863544 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:04:38.863659 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.863625 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.918223 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.918204 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:04:38.964411 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.964393 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:38.966539 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:38.966526 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:04:39.340246 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.340221 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 00:04:39.340792 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.340313 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 00:04:39.340792 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.340340 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 00:04:39.340792 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.340391 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 00:04:39.340792 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:39.340387 2571 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a94de1dd7bb8b42158c74d34dda92279-9029fc03a8717821.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-199.us-west-1.compute.internal.18e4602871d5b6f1\": read tcp 10.0.135.199:42608->18.189.122.36:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-199.us-west-1.compute.internal.18e4602871d5b6f1 default 5220 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-199.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-199.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-199.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 00:04:38 +0000 UTC,LastTimestamp:2026-11-03 00:04:38.640449078 +0000 UTC m=+0.558204377,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-199.us-west-1.compute.internal,}" Nov 03 00:04:39.382107 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.382086 2571 apiserver.go:52] "Watching apiserver" Nov 03 00:04:39.404332 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.404314 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 00:04:39.619743 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.619652 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 23:59:39 +0000 UTC" deadline="2028-07-02 08:10:49.878024224 +0000 UTC" Nov 03 00:04:39.619743 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.619700 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14576h6m10.258327893s" Nov 03 00:04:39.646082 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.646060 2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 00:04:39.646349 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.646333 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-nckb2","openshift-multus/network-metrics-daemon-67vkk","openshift-multus/multus-additional-cni-plugins-jkz9t","openshift-network-operator/iptables-alerter-7z644","openshift-ovn-kubernetes/ovnkube-node-g9lrv","kube-system/global-pull-secret-syncer-52cjz","kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal","openshift-multus/multus-ssc4h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx","openshift-cluster-node-tuning-operator/tuned-fnvz5","openshift-network-diagnostics/network-check-target-x6rmn","kube-system/konnectivity-agent-njmnx","openshift-dns/node-resolver-8cpwh"] Nov 03 00:04:39.648633 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.648616 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.650537 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.650522 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:04:39.652403 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.652391 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.654371 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.654358 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:04:39.656552 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.656531 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.658643 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.658630 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:04:39.660694 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.660662 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.663329 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.663313 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:39.663408 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:39.663392 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:04:39.665407 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.665393 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:39.665456 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:39.665443 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:04:39.667476 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.667461 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:04:39.669516 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.669504 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.671747 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.671733 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:39.671803 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:39.671785 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:04:39.701271 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.701238 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 00:04:39.701271 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.701264 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.701410 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.701282 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.701461 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.701410 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f67qw\"" Nov 03 00:04:39.702132 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.702116 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 00:04:39.702212 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.702155 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 00:04:39.702265 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.702214 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pkcxp\"" Nov 03 00:04:39.703802 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.703785 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.703883 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.703870 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 00:04:39.703962 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.703949 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.704006 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.703962 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g7949\"" Nov 03 00:04:39.704128 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.704115 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 00:04:39.704173 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.704160 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 00:04:39.705892 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.705880 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 00:04:39.712119 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712097 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.712205 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712133 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28lwp\" (UniqueName: \"kubernetes.io/projected/43785dfc-6e27-45e2-8608-5fec9dfa4a86-kube-api-access-28lwp\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.712205 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712159 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-netns\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.712205 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712175 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.712370 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712215 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddae2ef2-bac4-44c3-bef2-0d40c705c141-hosts-file\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:04:39.712370 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712242 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-cni-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.712370 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712266 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-conf-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.712370 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712294 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-node-log\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.712370 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712312 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4aa46084-b411-4ee1-98bb-145b0e8438c7-cni-binary-copy\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.712370 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712334 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-k8s-cni-cncf-io\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.712644 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712370 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-cni-bin\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.712644 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712400 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-systemd\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.712644 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712436 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-systemd\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.712644 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712463 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovnkube-config\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.712644 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712501 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8sg\" (UniqueName: \"kubernetes.io/projected/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-kube-api-access-jg8sg\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:39.712644 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712529 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58742d3c-ca4d-424e-aec7-35661094ad41-tmp\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.712644 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712552 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-var-lib-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.712644 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712579 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.712644 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712604 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.712644 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712620 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovn-node-metrics-cert\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.712644 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712642 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58742d3c-ca4d-424e-aec7-35661094ad41-etc-tuned\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712657 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712690 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddae2ef2-bac4-44c3-bef2-0d40c705c141-tmp-dir\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712713 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-system-cni-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712735 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-cni-multus\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712763 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-hostroot\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712797 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc96d\" (UniqueName: \"kubernetes.io/projected/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-kube-api-access-nc96d\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712837 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7314649e-0fd8-4bcd-bcfa-5140fb2711cf-konnectivity-ca\") pod \"konnectivity-agent-njmnx\" (UID: \"7314649e-0fd8-4bcd-bcfa-5140fb2711cf\") " pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712891 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-kubernetes\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712940 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj8fg\" (UniqueName: \"kubernetes.io/projected/5367d682-754b-4925-ae57-40a1246d15e2-kube-api-access-pj8fg\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712969 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-run-netns\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.712995 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-cni-bin\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713026 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-multus-certs\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713048 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysctl-conf\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713084 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-lib-modules\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713104 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.713167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713134 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-sys-fs\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713164 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cnibin\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713188 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-systemd-units\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713210 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-kubelet\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713254 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxj7\" (UniqueName: \"kubernetes.io/projected/4aa46084-b411-4ee1-98bb-145b0e8438c7-kube-api-access-wfxj7\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713311 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysconfig\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713332 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713380 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713409 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713434 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5367d682-754b-4925-ae57-40a1246d15e2-host-slash\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713457 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-host\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713485 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-run\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713506 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-sys\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713529 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-socket-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713557 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713580 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxrpd\" (UniqueName: \"kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd\") pod \"network-check-target-x6rmn\" (UID: \"fd761dae-da21-44e4-adad-287a5d204bd3\") " pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:39.713943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713618 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysctl-d\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713643 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fdf82aaa-e340-4a97-a3f4-297694eccab3-dbus\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713695 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-log-socket\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713729 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovnkube-script-lib\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713754 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-socket-dir-parent\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713779 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-daemon-config\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713804 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-serviceca\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713831 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-registration-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713856 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df88n\" (UniqueName: \"kubernetes.io/projected/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-kube-api-access-df88n\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713876 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-etc-kubernetes\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713889 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713904 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz5nz\" (UniqueName: \"kubernetes.io/projected/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-kube-api-access-tz5nz\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.713927 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5367d682-754b-4925-ae57-40a1246d15e2-iptables-alerter-script\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714075 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-etc-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714123 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-ovn\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714146 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-cni-netd\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.714634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714164 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-cnibin\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.715108 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714191 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7314649e-0fd8-4bcd-bcfa-5140fb2711cf-agent-certs\") pod \"konnectivity-agent-njmnx\" (UID: \"7314649e-0fd8-4bcd-bcfa-5140fb2711cf\") " pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:04:39.715108 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714228 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-var-lib-kubelet\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.715108 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714244 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-device-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.715108 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714266 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9pj8\" (UniqueName: \"kubernetes.io/projected/ddae2ef2-bac4-44c3-bef2-0d40c705c141-kube-api-access-z9pj8\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:04:39.715108 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714293 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-host\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.715108 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714311 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clcmb\" (UniqueName: \"kubernetes.io/projected/58742d3c-ca4d-424e-aec7-35661094ad41-kube-api-access-clcmb\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.715108 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714326 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fdf82aaa-e340-4a97-a3f4-297694eccab3-kubelet-config\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:39.715108 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714345 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-system-cni-dir\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.715108 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714360 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-kubelet\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.715108 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714386 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-env-overrides\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.715108 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714416 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-modprobe-d\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.715108 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.714436 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-slash\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.798965 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.798943 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 00:04:39.799055 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.798994 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9h9nq" Nov 03 00:04:39.801227 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801212 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 00:04:39.801360 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801345 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 00:04:39.801434 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801420 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-scmmg\"" Nov 03 00:04:39.801487 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801436 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 00:04:39.801647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801633 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nhzpn\"" Nov 03 00:04:39.801824 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801806 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.801876 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801861 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z6zhf\"" Nov 03 00:04:39.801876 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801864 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8glgl\"" Nov 03 00:04:39.801967 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801871 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.801967 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801888 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 00:04:39.801967 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801898 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.801967 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801901 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.801967 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801936 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.801967 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801916 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.801967 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.801902 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.813141 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.813125 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 00:04:39.813220 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.813180 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ptgp6\"" Nov 03 00:04:39.813220 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.813206 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 00:04:39.814660 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814641 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-cni-bin\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.814724 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814687 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-systemd\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.814724 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814706 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-systemd\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.814724 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814721 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovnkube-config\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.814836 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814734 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8sg\" (UniqueName: \"kubernetes.io/projected/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-kube-api-access-jg8sg\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:39.814836 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814738 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-cni-bin\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.814836 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814748 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58742d3c-ca4d-424e-aec7-35661094ad41-tmp\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.814836 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814791 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-var-lib-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.814836 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814818 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.814836 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814827 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-systemd\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.814836 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814837 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-var-lib-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814843 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814870 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovn-node-metrics-cert\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814885 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814894 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58742d3c-ca4d-424e-aec7-35661094ad41-etc-tuned\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814920 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814942 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddae2ef2-bac4-44c3-bef2-0d40c705c141-tmp-dir\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814902 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814962 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-system-cni-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814808 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-systemd\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.814999 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-cni-multus\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815007 2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815046 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-hostroot\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815072 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc96d\" (UniqueName: \"kubernetes.io/projected/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-kube-api-access-nc96d\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815094 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7314649e-0fd8-4bcd-bcfa-5140fb2711cf-konnectivity-ca\") pod \"konnectivity-agent-njmnx\" (UID: \"7314649e-0fd8-4bcd-bcfa-5140fb2711cf\") " pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815119 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-kubernetes\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815143 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj8fg\" (UniqueName: \"kubernetes.io/projected/5367d682-754b-4925-ae57-40a1246d15e2-kube-api-access-pj8fg\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:04:39.815160 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815166 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-run-netns\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815182 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-hostroot\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815185 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-cni-bin\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815230 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-cni-bin\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815248 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-multus-certs\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815275 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysctl-conf\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815300 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-lib-modules\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815344 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815369 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-sys-fs\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815395 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovnkube-config\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815407 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cnibin\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815433 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-systemd-units\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815459 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-kubelet\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815498 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxj7\" (UniqueName: \"kubernetes.io/projected/4aa46084-b411-4ee1-98bb-145b0e8438c7-kube-api-access-wfxj7\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815522 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysconfig\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815586 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815648 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.815998 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815698 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815725 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5367d682-754b-4925-ae57-40a1246d15e2-host-slash\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815772 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-host\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815793 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-kubernetes\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815797 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-run\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815836 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-sys\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815840 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7314649e-0fd8-4bcd-bcfa-5140fb2711cf-konnectivity-ca\") pod \"konnectivity-agent-njmnx\" (UID: \"7314649e-0fd8-4bcd-bcfa-5140fb2711cf\") " pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815861 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-socket-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815888 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815899 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-run-netns\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815914 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrpd\" (UniqueName: \"kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd\") pod \"network-check-target-x6rmn\" (UID: \"fd761dae-da21-44e4-adad-287a5d204bd3\") " pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815919 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-run\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815950 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815959 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-sys\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815938 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816022 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-systemd-units\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.815011 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-system-cni-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816042 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-socket-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.816889 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816062 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-multus-certs\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816111 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816145 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-cni-multus\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816204 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysctl-conf\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816323 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-lib-modules\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816348 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-kubelet\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816375 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816422 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-sys-fs\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816464 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cnibin\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816526 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816534 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-host\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816566 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysconfig\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816579 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5367d682-754b-4925-ae57-40a1246d15e2-host-slash\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816568 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysctl-d\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816613 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fdf82aaa-e340-4a97-a3f4-297694eccab3-dbus\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816635 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-log-socket\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:39.816647 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816659 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovnkube-script-lib\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.818175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816699 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-socket-dir-parent\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:39.816728 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret podName:fdf82aaa-e340-4a97-a3f4-297694eccab3 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:40.316708445 +0000 UTC m=+2.234463760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret") pod "global-pull-secret-syncer-52cjz" (UID: "fdf82aaa-e340-4a97-a3f4-297694eccab3") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816739 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-socket-dir-parent\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816748 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-daemon-config\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816804 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-log-socket\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816807 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-serviceca\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816836 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-registration-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816858 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df88n\" (UniqueName: \"kubernetes.io/projected/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-kube-api-access-df88n\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816873 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-etc-kubernetes\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816886 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816900 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz5nz\" (UniqueName: \"kubernetes.io/projected/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-kube-api-access-tz5nz\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816915 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5367d682-754b-4925-ae57-40a1246d15e2-iptables-alerter-script\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816930 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-etc-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816943 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-ovn\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816957 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-cni-netd\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816971 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-cnibin\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.819000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816984 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7314649e-0fd8-4bcd-bcfa-5140fb2711cf-agent-certs\") pod \"konnectivity-agent-njmnx\" (UID: \"7314649e-0fd8-4bcd-bcfa-5140fb2711cf\") " pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816997 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-var-lib-kubelet\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817012 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-device-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817025 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9pj8\" (UniqueName: \"kubernetes.io/projected/ddae2ef2-bac4-44c3-bef2-0d40c705c141-kube-api-access-z9pj8\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817040 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-host\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816774 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fdf82aaa-e340-4a97-a3f4-297694eccab3-dbus\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.816635 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysctl-d\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817125 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clcmb\" (UniqueName: \"kubernetes.io/projected/58742d3c-ca4d-424e-aec7-35661094ad41-kube-api-access-clcmb\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817150 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fdf82aaa-e340-4a97-a3f4-297694eccab3-kubelet-config\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817173 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-system-cni-dir\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817193 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-kubelet\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817216 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-host\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817219 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-env-overrides\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817229 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fdf82aaa-e340-4a97-a3f4-297694eccab3-kubelet-config\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817249 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-etc-kubernetes\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:39.817287 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817345 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817424 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddae2ef2-bac4-44c3-bef2-0d40c705c141-tmp-dir\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:04:39.819777 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817436 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-etc-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817473 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-cni-netd\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817488 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-var-lib-kubelet\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817495 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-device-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817191 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-registration-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817521 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-cnibin\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817527 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-ovn\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817553 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-kubelet\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817560 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-modprobe-d\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817590 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-system-cni-dir\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817592 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-slash\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:39.817614 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs podName:ea31369b-b8c9-4405-ab0c-6796cd41cd7a nodeName:}" failed. No retries permitted until 2026-11-03 00:04:40.317598007 +0000 UTC m=+2.235353294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs") pod "network-metrics-daemon-67vkk" (UID: "ea31369b-b8c9-4405-ab0c-6796cd41cd7a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817630 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-slash\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817659 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817705 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-modprobe-d\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817704 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lwp\" (UniqueName: \"kubernetes.io/projected/43785dfc-6e27-45e2-8608-5fec9dfa4a86-kube-api-access-28lwp\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.820647 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817739 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-netns\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817763 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817811 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddae2ef2-bac4-44c3-bef2-0d40c705c141-hosts-file\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817831 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-cni-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817860 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-conf-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817881 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-node-log\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817901 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4aa46084-b411-4ee1-98bb-145b0e8438c7-cni-binary-copy\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817927 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-k8s-cni-cncf-io\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817949 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-env-overrides\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.817976 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.818000 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-conf-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.818007 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-netns\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.818041 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-node-log\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.818043 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-k8s-cni-cncf-io\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.818075 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-cni-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.818077 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddae2ef2-bac4-44c3-bef2-0d40c705c141-hosts-file\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.818555 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-daemon-config\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.818612 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4aa46084-b411-4ee1-98bb-145b0e8438c7-cni-binary-copy\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.821388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.818851 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58742d3c-ca4d-424e-aec7-35661094ad41-tmp\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.822209 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.818878 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58742d3c-ca4d-424e-aec7-35661094ad41-etc-tuned\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.822209 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.818979 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-serviceca\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:04:39.822209 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.819183 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovn-node-metrics-cert\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.822209 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.819841 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.822209 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.821556 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7314649e-0fd8-4bcd-bcfa-5140fb2711cf-agent-certs\") pod \"konnectivity-agent-njmnx\" (UID: \"7314649e-0fd8-4bcd-bcfa-5140fb2711cf\") " pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:04:39.851096 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.851075 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 00:04:39.851210 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.851079 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k6628\"" Nov 03 00:04:39.851210 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.851120 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 00:04:39.851210 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.851168 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 00:04:39.857824 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.857808 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovnkube-script-lib\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.857930 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.857916 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5367d682-754b-4925-ae57-40a1246d15e2-iptables-alerter-script\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:04:39.871006 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:39.870958 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:39.871006 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:39.870984 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:39.871006 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:39.870997 2571 projected.go:196] Error preparing data for projected volume kube-api-access-cxrpd for pod openshift-network-diagnostics/network-check-target-x6rmn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:39.872272 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:39.871854 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd podName:fd761dae-da21-44e4-adad-287a5d204bd3 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:40.371831955 +0000 UTC m=+2.289587262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cxrpd" (UniqueName: "kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd") pod "network-check-target-x6rmn" (UID: "fd761dae-da21-44e4-adad-287a5d204bd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:39.873888 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.873399 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxj7\" (UniqueName: \"kubernetes.io/projected/4aa46084-b411-4ee1-98bb-145b0e8438c7-kube-api-access-wfxj7\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:04:39.873888 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.873760 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc96d\" (UniqueName: \"kubernetes.io/projected/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-kube-api-access-nc96d\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:04:39.873888 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.873838 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj8fg\" (UniqueName: \"kubernetes.io/projected/5367d682-754b-4925-ae57-40a1246d15e2-kube-api-access-pj8fg\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:04:39.877271 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.877249 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8sg\" (UniqueName: \"kubernetes.io/projected/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-kube-api-access-jg8sg\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:39.877552 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.877524 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df88n\" (UniqueName: \"kubernetes.io/projected/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-kube-api-access-df88n\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:39.877728 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.877626 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz5nz\" (UniqueName: \"kubernetes.io/projected/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-kube-api-access-tz5nz\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.877946 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.877926 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28lwp\" (UniqueName: \"kubernetes.io/projected/43785dfc-6e27-45e2-8608-5fec9dfa4a86-kube-api-access-28lwp\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:39.878856 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.878838 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clcmb\" (UniqueName: \"kubernetes.io/projected/58742d3c-ca4d-424e-aec7-35661094ad41-kube-api-access-clcmb\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:39.878943 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.878864 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9pj8\" (UniqueName: \"kubernetes.io/projected/ddae2ef2-bac4-44c3-bef2-0d40c705c141-kube-api-access-z9pj8\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:04:39.987343 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.987317 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:04:39.998215 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:39.998195 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:04:40.006642 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:40.006610 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf58af1ed7bc599c71fbbaa4e8065ef6.slice/crio-8ced7c0f225c21078f70aa0d93ae5c09b38e6e4ba40773e95b3e812d45405c33 WatchSource:0}: Error finding container 8ced7c0f225c21078f70aa0d93ae5c09b38e6e4ba40773e95b3e812d45405c33: Status 404 returned error can't find the container with id 8ced7c0f225c21078f70aa0d93ae5c09b38e6e4ba40773e95b3e812d45405c33 Nov 03 00:04:40.006889 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:40.006870 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35dc422a_ae29_4b68_b3fc_efd9b44fa2dd.slice/crio-df9902e0aac9259e1b20e7a2b76d199df705d2a00827f5821e434d7adc400151 WatchSource:0}: Error finding container df9902e0aac9259e1b20e7a2b76d199df705d2a00827f5821e434d7adc400151: Status 404 returned error can't find the container with id df9902e0aac9259e1b20e7a2b76d199df705d2a00827f5821e434d7adc400151 Nov 03 00:04:40.007319 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:40.007298 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf730cee51b9b033223ceb412560cad.slice/crio-7c1d91f20f7497418646f99c8223b2ad691358891c6cc6c73ae84e841d76e320 WatchSource:0}: Error finding container 7c1d91f20f7497418646f99c8223b2ad691358891c6cc6c73ae84e841d76e320: Status 404 returned error can't find the container with id 7c1d91f20f7497418646f99c8223b2ad691358891c6cc6c73ae84e841d76e320 Nov 03 00:04:40.011910 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:40.011871 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7314649e_0fd8_4bcd_bcfa_5140fb2711cf.slice/crio-c50d49b9f82cd3ed2eec19257ca8e031400c195d5cfb22bf22e8daf3206f5ed6 WatchSource:0}: Error finding container c50d49b9f82cd3ed2eec19257ca8e031400c195d5cfb22bf22e8daf3206f5ed6: Status 404 returned error can't find the container with id c50d49b9f82cd3ed2eec19257ca8e031400c195d5cfb22bf22e8daf3206f5ed6 Nov 03 00:04:40.012045 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:40.012031 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:04:40.028149 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:40.028130 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:04:40.033000 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:40.032987 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:04:40.037037 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:40.037020 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d41c02_03bc_4840_ae2f_a1590dda4ee9.slice/crio-b4a59b5627bd5b0f70bb3ad7d6fa09bcbf2054feed3c2ff050080564e3c924d3 WatchSource:0}: Error finding container b4a59b5627bd5b0f70bb3ad7d6fa09bcbf2054feed3c2ff050080564e3c924d3: Status 404 returned error can't find the container with id b4a59b5627bd5b0f70bb3ad7d6fa09bcbf2054feed3c2ff050080564e3c924d3 Nov 03 00:04:40.041725 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:40.041708 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5367d682_754b_4925_ae57_40a1246d15e2.slice/crio-cc06d0690d91342e97c77ca5c6a01f5c686d40166348547a5d77f4236f3497f4 WatchSource:0}: Error finding container cc06d0690d91342e97c77ca5c6a01f5c686d40166348547a5d77f4236f3497f4: Status 404 returned error can't find the container with id cc06d0690d91342e97c77ca5c6a01f5c686d40166348547a5d77f4236f3497f4 Nov 03 00:04:40.049809 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:40.049794 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:04:40.058015 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:40.057998 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43785dfc_6e27_45e2_8608_5fec9dfa4a86.slice/crio-67327af4b2141d01e717c838e10c38b51b72286ebf676a371297abe9bde098b3 WatchSource:0}: Error finding container 67327af4b2141d01e717c838e10c38b51b72286ebf676a371297abe9bde098b3: Status 404 returned error can't find the container with id 67327af4b2141d01e717c838e10c38b51b72286ebf676a371297abe9bde098b3 Nov 03 00:04:40.058797 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:40.058784 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:04:40.064648 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:40.064635 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ssc4h" Nov 03 00:04:40.066324 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:40.066308 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddae2ef2_bac4_44c3_bef2_0d40c705c141.slice/crio-ea7935e552c6446ca8941dff5f42f86474263ad910c86eed14b76f06739b0bbd WatchSource:0}: Error finding container ea7935e552c6446ca8941dff5f42f86474263ad910c86eed14b76f06739b0bbd: Status 404 returned error can't find the container with id ea7935e552c6446ca8941dff5f42f86474263ad910c86eed14b76f06739b0bbd Nov 03 00:04:40.073013 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:40.072994 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa46084_b411_4ee1_98bb_145b0e8438c7.slice/crio-3c70353946f43e9ce390ba9999746c9988a0b338c897bdfc31848c4e6c343335 WatchSource:0}: Error finding container 3c70353946f43e9ce390ba9999746c9988a0b338c897bdfc31848c4e6c343335: Status 404 returned error can't find the container with id 3c70353946f43e9ce390ba9999746c9988a0b338c897bdfc31848c4e6c343335 Nov 03 00:04:40.077217 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:40.077203 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:04:40.081700 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:40.081686 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:04:40.087745 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:40.087724 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be383eb_5b9b_4b12_b0ee_4dc8a04c02dc.slice/crio-45140effbc3f1d941815a354f960bc0cf50243c7d77daabf71b2cff913846c35 WatchSource:0}: Error finding container 45140effbc3f1d941815a354f960bc0cf50243c7d77daabf71b2cff913846c35: Status 404 returned error can't find the container with id 45140effbc3f1d941815a354f960bc0cf50243c7d77daabf71b2cff913846c35 Nov 03 00:04:40.091824 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:04:40.091806 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58742d3c_ca4d_424e_aec7_35661094ad41.slice/crio-efef8f1b5838e9824bdd9210339709e86737f8f6079c4c421c4f735b8f75bbe2 WatchSource:0}: Error finding container efef8f1b5838e9824bdd9210339709e86737f8f6079c4c421c4f735b8f75bbe2: Status 404 returned error can't find the container with id efef8f1b5838e9824bdd9210339709e86737f8f6079c4c421c4f735b8f75bbe2 Nov 03 00:04:40.093471 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:40.093455 2571 kuberuntime_manager.go:1707] "Unhandled Error" err="container tuned start failed in pod tuned-fnvz5_openshift-cluster-node-tuning-operator(58742d3c-ca4d-424e-aec7-35661094ad41): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Nov 03 00:04:40.093506 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:40.093483 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" podUID="58742d3c-ca4d-424e-aec7-35661094ad41" Nov 03 00:04:40.320982 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:40.320958 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:40.321121 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:40.320999 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:40.321121 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:40.321090 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:40.321121 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:40.321108 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:40.321242 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:40.321135 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs podName:ea31369b-b8c9-4405-ab0c-6796cd41cd7a nodeName:}" failed. No retries permitted until 2026-11-03 00:04:41.32112179 +0000 UTC m=+3.238877088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs") pod "network-metrics-daemon-67vkk" (UID: "ea31369b-b8c9-4405-ab0c-6796cd41cd7a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:40.321242 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:40.321163 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret podName:fdf82aaa-e340-4a97-a3f4-297694eccab3 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:41.32114612 +0000 UTC m=+3.238901422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret") pod "global-pull-secret-syncer-52cjz" (UID: "fdf82aaa-e340-4a97-a3f4-297694eccab3") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:40.421835 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:40.421799 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrpd\" (UniqueName: \"kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd\") pod \"network-check-target-x6rmn\" (UID: \"fd761dae-da21-44e4-adad-287a5d204bd3\") " pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:40.422426 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:40.421942 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:40.422426 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:40.421957 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:40.422426 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:40.421967 2571 projected.go:196] Error preparing data for projected volume kube-api-access-cxrpd for pod openshift-network-diagnostics/network-check-target-x6rmn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:40.422426 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:40.422016 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd podName:fd761dae-da21-44e4-adad-287a5d204bd3 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:41.421999271 +0000 UTC m=+3.339754571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cxrpd" (UniqueName: "kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd") pod "network-check-target-x6rmn" (UID: "fd761dae-da21-44e4-adad-287a5d204bd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:41.715430 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:41.715405 2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:41.758432 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:41.758379 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nckb2" event={"ID":"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc","Type":"ContainerStarted","Data":"45140effbc3f1d941815a354f960bc0cf50243c7d77daabf71b2cff913846c35"} Nov 03 00:04:41.759383 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:41.759355 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ssc4h" event={"ID":"4aa46084-b411-4ee1-98bb-145b0e8438c7","Type":"ContainerStarted","Data":"3c70353946f43e9ce390ba9999746c9988a0b338c897bdfc31848c4e6c343335"} Nov 03 00:04:41.760342 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:41.760316 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7z644" event={"ID":"5367d682-754b-4925-ae57-40a1246d15e2","Type":"ContainerStarted","Data":"cc06d0690d91342e97c77ca5c6a01f5c686d40166348547a5d77f4236f3497f4"} Nov 03 00:04:41.761351 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:41.761322 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jkz9t" event={"ID":"d5d41c02-03bc-4840-ae2f-a1590dda4ee9","Type":"ContainerStarted","Data":"b4a59b5627bd5b0f70bb3ad7d6fa09bcbf2054feed3c2ff050080564e3c924d3"} Nov 03 00:04:41.762351 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:41.762321 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" event={"ID":"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd","Type":"ContainerStarted","Data":"df9902e0aac9259e1b20e7a2b76d199df705d2a00827f5821e434d7adc400151"} Nov 03 00:04:41.763345 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:41.763326 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" event={"ID":"58742d3c-ca4d-424e-aec7-35661094ad41","Type":"ContainerStarted","Data":"efef8f1b5838e9824bdd9210339709e86737f8f6079c4c421c4f735b8f75bbe2"} Nov 03 00:04:41.764400 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:41.764382 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8cpwh" event={"ID":"ddae2ef2-bac4-44c3-bef2-0d40c705c141","Type":"ContainerStarted","Data":"ea7935e552c6446ca8941dff5f42f86474263ad910c86eed14b76f06739b0bbd"} Nov 03 00:04:41.764943 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:41.764919 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" podUID="58742d3c-ca4d-424e-aec7-35661094ad41" Nov 03 00:04:41.765516 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:41.765496 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" event={"ID":"43785dfc-6e27-45e2-8608-5fec9dfa4a86","Type":"ContainerStarted","Data":"67327af4b2141d01e717c838e10c38b51b72286ebf676a371297abe9bde098b3"} Nov 03 00:04:41.766486 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:41.766468 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-njmnx" event={"ID":"7314649e-0fd8-4bcd-bcfa-5140fb2711cf","Type":"ContainerStarted","Data":"c50d49b9f82cd3ed2eec19257ca8e031400c195d5cfb22bf22e8daf3206f5ed6"} Nov 03 00:04:41.767433 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:41.767416 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" event={"ID":"bdf730cee51b9b033223ceb412560cad","Type":"ContainerStarted","Data":"7c1d91f20f7497418646f99c8223b2ad691358891c6cc6c73ae84e841d76e320"} Nov 03 00:04:41.768372 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:41.768353 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" event={"ID":"cf58af1ed7bc599c71fbbaa4e8065ef6","Type":"ContainerStarted","Data":"8ced7c0f225c21078f70aa0d93ae5c09b38e6e4ba40773e95b3e812d45405c33"} Nov 03 00:04:42.016933 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:42.016813 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:40 +0000 UTC" deadline="2028-08-05 07:55:01.127814674 +0000 UTC" Nov 03 00:04:42.016933 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:42.016879 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15391h50m19.110941309s" Nov 03 00:04:42.149207 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:42.149178 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:42.173544 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:42.173508 2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:42.544646 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:42.544597 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:42.544927 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:42.544684 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:42.544927 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:42.544842 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:42.544927 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:42.544906 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs podName:ea31369b-b8c9-4405-ab0c-6796cd41cd7a nodeName:}" failed. No retries permitted until 2026-11-03 00:04:44.544886992 +0000 UTC m=+5.247490876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs") pod "network-metrics-daemon-67vkk" (UID: "ea31369b-b8c9-4405-ab0c-6796cd41cd7a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:42.545118 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:42.544965 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:42.545118 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:42.544996 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret podName:fdf82aaa-e340-4a97-a3f4-297694eccab3 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:44.544986046 +0000 UTC m=+5.247589927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret") pod "global-pull-secret-syncer-52cjz" (UID: "fdf82aaa-e340-4a97-a3f4-297694eccab3") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:42.645571 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:42.645528 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrpd\" (UniqueName: \"kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd\") pod \"network-check-target-x6rmn\" (UID: \"fd761dae-da21-44e4-adad-287a5d204bd3\") " pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:42.646276 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:42.645715 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:42.646276 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:42.645736 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:42.646276 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:42.645752 2571 projected.go:196] Error preparing data for projected volume kube-api-access-cxrpd for pod openshift-network-diagnostics/network-check-target-x6rmn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:42.646276 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:42.645834 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd podName:fd761dae-da21-44e4-adad-287a5d204bd3 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:44.64579921 +0000 UTC m=+5.348403085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cxrpd" (UniqueName: "kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd") pod "network-check-target-x6rmn" (UID: "fd761dae-da21-44e4-adad-287a5d204bd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:42.755281 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:42.755244 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:42.755469 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:42.755385 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:04:42.755537 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:42.755495 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:42.755590 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:42.755574 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:04:42.755644 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:42.755633 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:42.755725 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:42.755707 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:04:43.018045 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:43.017932 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:40 +0000 UTC" deadline="2028-07-10 07:04:55.37931669 +0000 UTC" Nov 03 00:04:43.018045 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:43.018026 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14767h0m12.361321075s" Nov 03 00:04:44.564338 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:44.563438 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:44.564338 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:44.563753 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:44.564338 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:44.563916 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:44.564338 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:44.563984 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs podName:ea31369b-b8c9-4405-ab0c-6796cd41cd7a nodeName:}" failed. No retries permitted until 2026-11-03 00:04:48.563961612 +0000 UTC m=+9.266565499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs") pod "network-metrics-daemon-67vkk" (UID: "ea31369b-b8c9-4405-ab0c-6796cd41cd7a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:44.564338 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:44.564044 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:44.564338 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:44.564078 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret podName:fdf82aaa-e340-4a97-a3f4-297694eccab3 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:48.56406661 +0000 UTC m=+9.266670478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret") pod "global-pull-secret-syncer-52cjz" (UID: "fdf82aaa-e340-4a97-a3f4-297694eccab3") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:44.665545 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:44.665120 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrpd\" (UniqueName: \"kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd\") pod \"network-check-target-x6rmn\" (UID: \"fd761dae-da21-44e4-adad-287a5d204bd3\") " pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:44.665545 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:44.665305 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:44.665545 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:44.665329 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:44.665545 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:44.665342 2571 projected.go:196] Error preparing data for projected volume kube-api-access-cxrpd for pod openshift-network-diagnostics/network-check-target-x6rmn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:44.665545 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:44.665404 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd podName:fd761dae-da21-44e4-adad-287a5d204bd3 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:48.66538316 +0000 UTC m=+9.367987044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cxrpd" (UniqueName: "kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd") pod "network-check-target-x6rmn" (UID: "fd761dae-da21-44e4-adad-287a5d204bd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:44.754857 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:44.754409 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:44.754857 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:44.754590 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:04:44.754857 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:44.754662 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:44.754857 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:44.754713 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:04:44.754857 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:44.754757 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:44.754857 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:44.754805 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:04:44.798852 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:44.798468 2571 generic.go:356] "Generic (PLEG): container finished" podID="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" containerID="ee8bce2c98bd0bb07d12e1315603b8ecc7cf683e53cf32ebce911934fb47051d" exitCode=0 Nov 03 00:04:44.798852 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:44.798529 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jkz9t" event={"ID":"d5d41c02-03bc-4840-ae2f-a1590dda4ee9","Type":"ContainerDied","Data":"ee8bce2c98bd0bb07d12e1315603b8ecc7cf683e53cf32ebce911934fb47051d"} Nov 03 00:04:45.804664 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:45.804619 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-njmnx" event={"ID":"7314649e-0fd8-4bcd-bcfa-5140fb2711cf","Type":"ContainerStarted","Data":"a47f0ac7b9063f8d9344ed076ae7ec7cae27acdca9b2788482e43bfb1c5efabb"} Nov 03 00:04:45.810284 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:45.810246 2571 generic.go:356] "Generic (PLEG): container finished" podID="bdf730cee51b9b033223ceb412560cad" containerID="56df28890ee2e20e4617ba3c42b659c9055b084946299d3a316b187fd8160021" exitCode=0 Nov 03 00:04:45.810435 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:45.810303 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" event={"ID":"bdf730cee51b9b033223ceb412560cad","Type":"ContainerDied","Data":"56df28890ee2e20e4617ba3c42b659c9055b084946299d3a316b187fd8160021"} Nov 03 00:04:45.919084 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:45.918882 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-njmnx" podStartSLOduration=2.712429123 podStartE2EDuration="6.918862249s" totalImagesPullingTime="4.206433126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:45.91808362 +0000 UTC m=+6.620687510" watchObservedRunningTime="2026-11-03 00:04:45.918862249 +0000 UTC m=+6.621466137" Nov 03 00:04:46.247212 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:46.247131 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:04:46.247960 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:46.247934 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:04:46.754594 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:46.754565 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:46.754773 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:46.754691 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:04:46.754773 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:46.754760 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:46.754929 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:46.754856 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:04:46.754989 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:46.754933 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:46.755038 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:46.755021 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:04:46.817175 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:46.817139 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" event={"ID":"bdf730cee51b9b033223ceb412560cad","Type":"ContainerStarted","Data":"9c282d77a4d35a470acb2347fcd5d16c785952f2371ec8bff55e3dbad13dde96"} Nov 03 00:04:46.821848 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:46.821337 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" event={"ID":"cf58af1ed7bc599c71fbbaa4e8065ef6","Type":"ContainerStarted","Data":"76e4943e4878c599f3f55aea2fef2f8b5268bf67a9bb8afaebfd070560289cbd"} Nov 03 00:04:47.044034 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:47.042439 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" podStartSLOduration=7.042420599 podStartE2EDuration="7.042420599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:46.93024351 +0000 UTC m=+7.632847415" watchObservedRunningTime="2026-11-03 00:04:47.042420599 +0000 UTC m=+7.745024489" Nov 03 00:04:47.842867 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:47.838075 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nckb2" event={"ID":"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc","Type":"ContainerStarted","Data":"3a4545836b1c6233d6c968f033c142f986572a6522fa8bfdd4ef125a0297cd93"} Nov 03 00:04:47.842867 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:47.842451 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" event={"ID":"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd","Type":"ContainerStarted","Data":"2bdc8349c92a89285006734aac9c86f18238e4ac535342a3a5483e1a4ac4ec7d"} Nov 03 00:04:47.842867 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:47.842529 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:04:47.954130 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:47.953472 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" podStartSLOduration=7.953451151 podStartE2EDuration="7.953451151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:47.043637665 +0000 UTC m=+7.746241571" watchObservedRunningTime="2026-11-03 00:04:47.953451151 +0000 UTC m=+8.656055042" Nov 03 00:04:48.601130 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:48.600709 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:48.601130 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:48.600777 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:48.601130 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:48.600943 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:48.601130 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:48.601004 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs podName:ea31369b-b8c9-4405-ab0c-6796cd41cd7a nodeName:}" failed. No retries permitted until 2026-11-03 00:04:56.600984917 +0000 UTC m=+17.303588792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs") pod "network-metrics-daemon-67vkk" (UID: "ea31369b-b8c9-4405-ab0c-6796cd41cd7a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:48.601130 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:48.601067 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:48.601130 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:48.601098 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret podName:fdf82aaa-e340-4a97-a3f4-297694eccab3 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:56.6010881 +0000 UTC m=+17.303691980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret") pod "global-pull-secret-syncer-52cjz" (UID: "fdf82aaa-e340-4a97-a3f4-297694eccab3") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:48.701305 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:48.701241 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrpd\" (UniqueName: \"kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd\") pod \"network-check-target-x6rmn\" (UID: \"fd761dae-da21-44e4-adad-287a5d204bd3\") " pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:48.701412 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:48.701372 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:48.701412 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:48.701386 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:48.701412 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:48.701394 2571 projected.go:196] Error preparing data for projected volume kube-api-access-cxrpd for pod openshift-network-diagnostics/network-check-target-x6rmn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:48.701566 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:48.701432 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd podName:fd761dae-da21-44e4-adad-287a5d204bd3 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:56.701420079 +0000 UTC m=+17.404023959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cxrpd" (UniqueName: "kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd") pod "network-check-target-x6rmn" (UID: "fd761dae-da21-44e4-adad-287a5d204bd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:48.755100 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:48.755064 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:48.755262 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:48.755221 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:04:48.755324 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:48.755306 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:48.755422 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:48.755375 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:04:48.755422 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:48.755398 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:48.755526 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:48.755457 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:04:49.264547 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:49.264515 2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 00:04:49.715914 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:49.715800 2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:04:49.264536449Z","UUID":"c1344054-a6c0-448b-bd91-168fa0c81cc4","Handler":null,"Name":"","Endpoint":""} Nov 03 00:04:49.717372 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:49.717350 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 00:04:49.717372 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:49.717382 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 00:04:49.854782 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:49.854748 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" event={"ID":"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd","Type":"ContainerStarted","Data":"cbf77935792f03174e10d574c9268b53e0f4cf4fdddd0de8b8bedd0f9348b079"} Nov 03 00:04:49.856085 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:49.856060 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8cpwh" event={"ID":"ddae2ef2-bac4-44c3-bef2-0d40c705c141","Type":"ContainerStarted","Data":"ca6ff2ace8c154917a20e0fb6808c8ffcffd114d9233c391dfbd857bebed4781"} Nov 03 00:04:49.857415 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:49.857387 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ssc4h" event={"ID":"4aa46084-b411-4ee1-98bb-145b0e8438c7","Type":"ContainerStarted","Data":"7216bc6427aa037bd8f3ae3d58e4865d711b276c597cff14b20b865872147cb4"} Nov 03 00:04:49.858991 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:49.858967 2571 generic.go:356] "Generic (PLEG): container finished" podID="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" containerID="e92ca27e33798c244770b8fea0938d82fe39e56df0c27929301ad0d0f283522d" exitCode=0 Nov 03 00:04:49.859092 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:49.858998 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jkz9t" event={"ID":"d5d41c02-03bc-4840-ae2f-a1590dda4ee9","Type":"ContainerDied","Data":"e92ca27e33798c244770b8fea0938d82fe39e56df0c27929301ad0d0f283522d"} Nov 03 00:04:49.966954 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:49.966858 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-8cpwh" podStartSLOduration=3.281477315 podStartE2EDuration="10.966843799s" totalImagesPullingTime="7.685366484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:49.966804699 +0000 UTC m=+10.669408606" watchObservedRunningTime="2026-11-03 00:04:49.966843799 +0000 UTC m=+10.669447688" Nov 03 00:04:49.967075 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:49.966991 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nckb2" podStartSLOduration=5.404971778 podStartE2EDuration="10.966982968s" totalImagesPullingTime="5.56201119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:47.953903422 +0000 UTC m=+8.656507327" watchObservedRunningTime="2026-11-03 00:04:49.966982968 +0000 UTC m=+10.669586856" Nov 03 00:04:50.196835 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:50.196769 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ssc4h" podStartSLOduration=3.711866503 podStartE2EDuration="11.196754142s" totalImagesPullingTime="7.484887639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:50.080731461 +0000 UTC m=+10.783335351" watchObservedRunningTime="2026-11-03 00:04:50.196754142 +0000 UTC m=+10.899358030" Nov 03 00:04:50.754659 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:50.754444 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:50.754957 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:50.754506 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:50.754957 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:50.754514 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:50.754957 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:50.754739 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:04:50.754957 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:50.754855 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:04:50.754957 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:50.754893 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:04:50.862374 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:50.862313 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7z644" event={"ID":"5367d682-754b-4925-ae57-40a1246d15e2","Type":"ContainerStarted","Data":"343f80cce9526e62b5d2f8db3d38ba42f8c985cbf12cb1c091bd5ace1ebe7825"} Nov 03 00:04:50.864837 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:50.864788 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" event={"ID":"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd","Type":"ContainerStarted","Data":"b2663cf39c45b08ea07be5b947e986295143e434ef58c700d77656e462432cb8"} Nov 03 00:04:51.088993 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:51.088940 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" podStartSLOduration=2.958757151 podStartE2EDuration="12.088920549s" totalImagesPullingTime="9.130163398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:51.088685908 +0000 UTC m=+11.791289775" watchObservedRunningTime="2026-11-03 00:04:51.088920549 +0000 UTC m=+11.791524439" Nov 03 00:04:51.089529 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:51.089489 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7z644" podStartSLOduration=4.670986982 podStartE2EDuration="12.089478029s" totalImagesPullingTime="7.418491047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:50.975153265 +0000 UTC m=+11.677757172" watchObservedRunningTime="2026-11-03 00:04:51.089478029 +0000 UTC m=+11.792081934" Nov 03 00:04:51.869554 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:51.869510 2571 generic.go:356] "Generic (PLEG): container finished" podID="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" containerID="3d389ef9792e057fd1631434e746945c57aecb79e92c1b0dcc5e7e74718688d9" exitCode=0 Nov 03 00:04:51.870012 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:51.869596 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jkz9t" event={"ID":"d5d41c02-03bc-4840-ae2f-a1590dda4ee9","Type":"ContainerDied","Data":"3d389ef9792e057fd1631434e746945c57aecb79e92c1b0dcc5e7e74718688d9"} Nov 03 00:04:52.755377 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:52.755344 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:52.755377 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:52.755348 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:52.755684 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:52.755526 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:04:52.755684 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:52.755589 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:52.755779 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:52.755667 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:04:52.755779 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:52.755712 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:04:53.876689 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:53.876639 2571 generic.go:356] "Generic (PLEG): container finished" podID="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" containerID="2940ea85428cc337bf03b9b330e4e7b995b1b7d0b8edf296b2f5314cf568ab94" exitCode=0 Nov 03 00:04:53.877405 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:53.876719 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jkz9t" event={"ID":"d5d41c02-03bc-4840-ae2f-a1590dda4ee9","Type":"ContainerDied","Data":"2940ea85428cc337bf03b9b330e4e7b995b1b7d0b8edf296b2f5314cf568ab94"} Nov 03 00:04:54.754713 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:54.754674 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:54.754936 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:54.754797 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:54.754936 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:54.754807 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:04:54.755046 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:54.754942 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:04:54.755046 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:54.755011 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:54.755144 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:54.755095 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:04:56.661674 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:56.661456 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:56.662074 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:56.661653 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:56.662074 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:56.661806 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret podName:fdf82aaa-e340-4a97-a3f4-297694eccab3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.661781802 +0000 UTC m=+33.364385681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret") pod "global-pull-secret-syncer-52cjz" (UID: "fdf82aaa-e340-4a97-a3f4-297694eccab3") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:56.662074 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:56.661816 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:56.662074 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:56.661887 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs podName:ea31369b-b8c9-4405-ab0c-6796cd41cd7a nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.661870452 +0000 UTC m=+33.364474321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs") pod "network-metrics-daemon-67vkk" (UID: "ea31369b-b8c9-4405-ab0c-6796cd41cd7a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:56.662074 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:56.661731 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:56.755638 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:56.755232 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:56.755638 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:56.755358 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:04:56.755638 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:56.755417 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:56.755638 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:56.755470 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:04:56.755638 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:56.755513 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:56.755638 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:56.755594 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:04:56.763145 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:56.763057 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrpd\" (UniqueName: \"kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd\") pod \"network-check-target-x6rmn\" (UID: \"fd761dae-da21-44e4-adad-287a5d204bd3\") " pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:56.763304 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:56.763239 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:56.763304 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:56.763272 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:56.763304 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:56.763285 2571 projected.go:196] Error preparing data for projected volume kube-api-access-cxrpd for pod openshift-network-diagnostics/network-check-target-x6rmn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:56.763455 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:56.763335 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd podName:fd761dae-da21-44e4-adad-287a5d204bd3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.76332269 +0000 UTC m=+33.465926581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cxrpd" (UniqueName: "kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd") pod "network-check-target-x6rmn" (UID: "fd761dae-da21-44e4-adad-287a5d204bd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:56.887470 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:56.887394 2571 generic.go:356] "Generic (PLEG): container finished" podID="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" containerID="78cd4792fe2c55b193e89938eebbee01eda398559e183d0db2a74de260020a97" exitCode=0 Nov 03 00:04:56.887470 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:56.887424 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jkz9t" event={"ID":"d5d41c02-03bc-4840-ae2f-a1590dda4ee9","Type":"ContainerDied","Data":"78cd4792fe2c55b193e89938eebbee01eda398559e183d0db2a74de260020a97"} Nov 03 00:04:57.893043 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:57.893012 2571 generic.go:356] "Generic (PLEG): container finished" podID="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" containerID="8ad37539a5307bb9b865264eb9949f64cf70e137c25999c1d707e92b19fc5870" exitCode=0 Nov 03 00:04:57.893472 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:57.893051 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jkz9t" event={"ID":"d5d41c02-03bc-4840-ae2f-a1590dda4ee9","Type":"ContainerDied","Data":"8ad37539a5307bb9b865264eb9949f64cf70e137c25999c1d707e92b19fc5870"} Nov 03 00:04:58.660874 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:58.660789 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:58.755268 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:58.755234 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:04:58.755268 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:58.755256 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:04:58.755494 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:58.755256 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:04:58.755494 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:58.755370 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:04:58.755494 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:58.755423 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:04:58.755625 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:04:58.755507 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:04:58.897385 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:58.896836 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" event={"ID":"58742d3c-ca4d-424e-aec7-35661094ad41","Type":"ContainerStarted","Data":"affdbaf6d14a9d65caa40e44c057c1ff9ba6fa65eea303762779d71bd7df83ab"} Nov 03 00:04:58.900748 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:58.900712 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jkz9t" event={"ID":"d5d41c02-03bc-4840-ae2f-a1590dda4ee9","Type":"ContainerStarted","Data":"44161b07801988360e513f0f81ecd26b948868be7b99020fa4f2428f885c01d5"} Nov 03 00:04:58.926862 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:58.926772 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:59.011329 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:59.011282 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" podStartSLOduration=2.937346683 podStartE2EDuration="20.011264185s" totalImagesPullingTime="17.073917502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-11-03 00:04:59.010906738 +0000 UTC m=+19.713510655" watchObservedRunningTime="2026-11-03 00:04:59.011264185 +0000 UTC m=+19.713868075" Nov 03 00:04:59.128550 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:59.128493 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jkz9t" podStartSLOduration=7.77248549 podStartE2EDuration="20.128473932s" totalImagesPullingTime="12.355988442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:59.127699861 +0000 UTC m=+19.830303750" watchObservedRunningTime="2026-11-03 00:04:59.128473932 +0000 UTC m=+19.831077820" Nov 03 00:04:59.132967 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:59.132937 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:59.290068 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:59.290041 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:59.906866 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:59.906656 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" event={"ID":"43785dfc-6e27-45e2-8608-5fec9dfa4a86","Type":"ContainerStarted","Data":"0a103d17d6aa5240429521f426fdafc155ccd576162038601c2f7af5dfc47a91"} Nov 03 00:04:59.907177 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:59.906942 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" event={"ID":"43785dfc-6e27-45e2-8608-5fec9dfa4a86","Type":"ContainerStarted","Data":"f8e13becc4f815818dbf675900d5810da1e8b16b79111ba6cfca630ff6cce16a"} Nov 03 00:04:59.907177 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:04:59.906980 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" event={"ID":"43785dfc-6e27-45e2-8608-5fec9dfa4a86","Type":"ContainerStarted","Data":"15e64f88131b430ede7172f1dc18bd4b27b880d859f822c8e335f424b662e98a"} Nov 03 00:05:00.606010 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:00.605972 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:00.754737 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:00.754707 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:00.754906 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:00.754802 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:00.754906 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:00.754807 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:00.754969 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:00.754913 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:00.754969 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:00.754938 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:00.755025 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:00.754997 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:00.910739 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:00.910674 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" event={"ID":"43785dfc-6e27-45e2-8608-5fec9dfa4a86","Type":"ContainerStarted","Data":"01888cf883bac93eb7de143c696927baab321c069cc55b5c35e8462cbd798bc1"} Nov 03 00:05:00.910739 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:00.910737 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" event={"ID":"43785dfc-6e27-45e2-8608-5fec9dfa4a86","Type":"ContainerStarted","Data":"4e39665e4b40882c00a6aec692b77725460be04e859dee81e4edba2d1b640fd8"} Nov 03 00:05:00.911105 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:00.910768 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" event={"ID":"43785dfc-6e27-45e2-8608-5fec9dfa4a86","Type":"ContainerStarted","Data":"d898d626e0609cfecbca2697da9388c6f38ac40eef5dd6a4789118b87eaf1484"} Nov 03 00:05:01.383169 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:01.383142 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:05:01.383345 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:01.383258 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:05:01.383754 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:01.383736 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:05:01.645262 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:01.645179 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:01.803363 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:01.803337 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:02.754934 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:02.754895 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:02.755367 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:02.754949 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:02.755367 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:02.754905 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:02.755367 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:02.755023 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:02.755367 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:02.755065 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:02.755367 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:02.755144 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:02.920646 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:02.920616 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" event={"ID":"43785dfc-6e27-45e2-8608-5fec9dfa4a86","Type":"ContainerStarted","Data":"545e98dfedca6bb773076b0de6e862d3076ad666f53b68f363416492145f4631"} Nov 03 00:05:03.770181 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:03.770144 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:04.755039 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:04.755012 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:04.755195 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:04.755086 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:04.755195 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:04.755171 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:04.755195 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:04.755182 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:04.755288 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:04.755228 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:04.755338 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:04.755316 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:04.857210 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:04.857181 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:04.927299 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:04.927273 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" event={"ID":"43785dfc-6e27-45e2-8608-5fec9dfa4a86","Type":"ContainerStarted","Data":"b561fbc0cdf94f764ef4f5e71561d53ed15acf8aeb7e92da0196c81b743bc787"} Nov 03 00:05:04.927536 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:04.927521 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:04.927617 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:04.927587 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:04.942802 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:04.942784 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:05.131977 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:05.131950 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:05.200980 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:05.200936 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" podStartSLOduration=7.712729391 podStartE2EDuration="26.200924772s" totalImagesPullingTime="18.488195381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:05.148808991 +0000 UTC m=+25.851412879" watchObservedRunningTime="2026-11-03 00:05:05.200924772 +0000 UTC m=+25.903528686" Nov 03 00:05:05.332388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:05.332363 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:05.486445 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:05.486371 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:05.933409 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:05.933367 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:05.948398 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:05.948367 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:06.134058 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:06.134026 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:06.398105 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:06.398076 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:06.610397 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:06.610233 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:06.754977 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:06.754916 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:06.755137 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:06.755011 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:06.755137 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:06.755035 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:06.755211 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:06.755163 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:06.755293 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:06.755276 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:06.755372 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:06.755357 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:06.890759 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:06.890724 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x6rmn"] Nov 03 00:05:06.893616 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:06.893591 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-67vkk"] Nov 03 00:05:06.894352 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:06.894333 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-52cjz"] Nov 03 00:05:06.934161 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:06.934139 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:06.934161 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:06.934151 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:06.934592 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:06.934140 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:06.934592 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:06.934257 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:06.934592 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:06.934326 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:06.934705 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:06.934641 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:07.367223 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:07.367185 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:07.536360 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:07.536330 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:08.048992 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:08.048962 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:08.087221 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:08.087193 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8cpwh_ddae2ef2-bac4-44c3-bef2-0d40c705c141/dns-node-resolver/0.log" Nov 03 00:05:08.754552 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:08.754448 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:08.754552 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:08.754491 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:08.754762 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:08.754461 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:08.754762 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:08.754564 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:08.754762 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:08.754639 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:08.754762 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:08.754688 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:09.280703 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:09.280678 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nckb2_4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc/node-ca/0.log" Nov 03 00:05:09.289388 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:09.289365 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:09.446908 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:09.446867 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:09.750624 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:09.750588 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:09.908650 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:09.908621 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:10.409094 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:10.409059 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:10.648306 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:10.648276 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:10.754608 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:10.754576 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:10.754767 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:10.754619 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:10.754767 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:10.754641 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:10.754767 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:10.754735 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:10.754883 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:10.754868 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:10.754955 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:10.754941 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:11.021146 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:11.021069 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:11.227980 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:11.227944 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:11.412368 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:11.412338 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:11.618311 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:11.618275 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:11.813077 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:11.813038 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:12.202517 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:12.202439 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:12.684057 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:12.684026 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:12.684442 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:12.684090 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:12.684442 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:12.684169 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:05:12.684442 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:12.684169 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:05:12.684442 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:12.684224 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs podName:ea31369b-b8c9-4405-ab0c-6796cd41cd7a nodeName:}" failed. No retries permitted until 2026-11-03 00:05:44.684210092 +0000 UTC m=+65.386813963 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs") pod "network-metrics-daemon-67vkk" (UID: "ea31369b-b8c9-4405-ab0c-6796cd41cd7a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:05:12.684442 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:12.684242 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret podName:fdf82aaa-e340-4a97-a3f4-297694eccab3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:44.684235772 +0000 UTC m=+65.386839641 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret") pod "global-pull-secret-syncer-52cjz" (UID: "fdf82aaa-e340-4a97-a3f4-297694eccab3") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:05:12.754868 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:12.754841 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:12.755033 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:12.754927 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:12.755033 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:12.754930 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:12.755033 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:12.754933 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:12.755033 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:12.755015 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:12.755207 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:12.755108 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:12.785067 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:12.785039 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrpd\" (UniqueName: \"kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd\") pod \"network-check-target-x6rmn\" (UID: \"fd761dae-da21-44e4-adad-287a5d204bd3\") " pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:12.785200 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:12.785178 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:05:12.785200 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:12.785193 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:05:12.785261 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:12.785202 2571 projected.go:196] Error preparing data for projected volume kube-api-access-cxrpd for pod openshift-network-diagnostics/network-check-target-x6rmn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:05:12.785261 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:12.785247 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd podName:fd761dae-da21-44e4-adad-287a5d204bd3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:44.78523137 +0000 UTC m=+65.487835251 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cxrpd" (UniqueName: "kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd") pod "network-check-target-x6rmn" (UID: "fd761dae-da21-44e4-adad-287a5d204bd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:05:12.937361 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:12.937284 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:13.126077 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:13.126039 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:13.288572 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:13.288539 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:13.466839 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:13.466786 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:13.657880 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:13.657778 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:13.826434 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:13.826395 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:14.044017 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:14.043986 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:14.209741 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:14.209715 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:14.363434 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:14.363359 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:14.754677 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:14.754599 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:14.754677 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:14.754670 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:14.754888 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:14.754699 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:14.754888 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:14.754780 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:14.754888 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:14.754833 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:14.754994 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:14.754909 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:15.009911 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:15.009814 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:15.200176 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:15.200142 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:15.421356 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:15.421321 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:15.607266 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:15.607234 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:15.812625 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:15.812594 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:16.205128 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:16.205060 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:16.754371 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:16.754340 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:16.754543 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:16.754340 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:16.754543 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:16.754444 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:16.754543 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:16.754340 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:16.754543 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:16.754528 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:16.754680 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:16.754605 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:17.011140 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:17.011111 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:17.169646 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:17.169605 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:18.754768 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:18.754736 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:18.755218 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:18.754806 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:18.755218 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:18.754908 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:18.755218 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:18.754922 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:18.755218 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:18.755003 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:18.755218 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:18.755093 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:20.754561 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:20.754534 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:20.754561 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:20.754559 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:20.754974 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:20.754559 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:20.754974 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:20.754627 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:20.754974 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:20.754705 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:20.754974 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:20.754772 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:22.754635 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:22.754600 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:22.754635 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:22.754641 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:22.755081 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:22.754600 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:22.755081 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:22.754698 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:22.755081 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:22.754789 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:22.755081 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:22.754868 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:23.841905 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:23.841872 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:24.107112 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:24.107037 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:24.754946 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:24.754914 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:24.754946 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:24.754945 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:24.755156 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:24.755030 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:24.755156 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:24.755030 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:24.755156 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:24.755112 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:24.755265 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:24.755181 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:24.881919 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:24.881889 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:25.132033 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:25.131999 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:26.262374 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:26.262341 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:26.418876 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:26.418841 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:26.754869 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:26.754841 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:26.754869 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:26.754850 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:26.755080 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:26.754935 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:26.755080 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:26.755012 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:26.755080 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:26.755068 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:26.755180 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:26.755124 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:28.691382 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:28.691350 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:28.754506 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:28.754481 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:28.754649 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:28.754519 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:28.754649 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:28.754582 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:28.754649 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:28.754597 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52cjz" podUID="fdf82aaa-e340-4a97-a3f4-297694eccab3" Nov 03 00:05:28.754737 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:28.754656 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-67vkk" podUID="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" Nov 03 00:05:28.754737 ip-10-0-135-199 kubenswrapper[2571]: E1103 00:05:28.754714 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6rmn" podUID="fd761dae-da21-44e4-adad-287a5d204bd3" Nov 03 00:05:28.862335 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:28.862305 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:29.193633 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.193608 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:29.354732 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.354696 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:29.547281 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.547252 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:29.657288 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.657266 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeReady" Nov 03 00:05:29.657413 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.657360 2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 00:05:29.811517 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.811491 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rwmxk"] Nov 03 00:05:29.813292 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.813273 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cdsd8"] Nov 03 00:05:29.813406 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.813395 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:29.815039 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.815022 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cdsd8" Nov 03 00:05:29.819077 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.819056 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6v6bq"] Nov 03 00:05:29.820709 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.820687 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rwmxk"] Nov 03 00:05:29.820850 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.820738 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cdsd8"] Nov 03 00:05:29.820850 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.820761 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:29.834257 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.834239 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6v6bq"] Nov 03 00:05:29.866903 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.866885 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 00:05:29.866903 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.866893 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 00:05:29.866903 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.866902 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 00:05:29.867063 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.866892 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bkhng\"" Nov 03 00:05:29.869339 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.869316 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 00:05:29.869426 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.869366 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 00:05:29.869426 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.869422 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jkmg8\"" Nov 03 00:05:29.872744 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.872727 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 00:05:29.872867 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.872740 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 00:05:29.872867 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.872763 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q5tjj\"" Nov 03 00:05:29.911870 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.911850 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b962e9e5-781e-450c-8c3f-17e217c67df6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:29.911981 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.911876 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp5bx\" (UniqueName: \"kubernetes.io/projected/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0-kube-api-access-gp5bx\") pod \"ingress-canary-cdsd8\" (UID: \"6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0\") " pod="openshift-ingress-canary/ingress-canary-cdsd8" Nov 03 00:05:29.911981 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.911896 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b962e9e5-781e-450c-8c3f-17e217c67df6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:29.911981 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.911911 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0-cert\") pod \"ingress-canary-cdsd8\" (UID: \"6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0\") " pod="openshift-ingress-canary/ingress-canary-cdsd8" Nov 03 00:05:29.912091 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.911983 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b962e9e5-781e-450c-8c3f-17e217c67df6-crio-socket\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:29.912091 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.912063 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86519442-21ae-43dc-98bc-730f8212e34b-config-volume\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:29.912169 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.912089 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86519442-21ae-43dc-98bc-730f8212e34b-metrics-tls\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:29.912169 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.912105 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86519442-21ae-43dc-98bc-730f8212e34b-tmp-dir\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:29.912261 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.912202 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b962e9e5-781e-450c-8c3f-17e217c67df6-data-volume\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:29.912300 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.912279 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95txv\" (UniqueName: \"kubernetes.io/projected/86519442-21ae-43dc-98bc-730f8212e34b-kube-api-access-95txv\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:29.912334 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.912307 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqgv8\" (UniqueName: \"kubernetes.io/projected/b962e9e5-781e-450c-8c3f-17e217c67df6-kube-api-access-sqgv8\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:29.967570 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.967546 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 00:05:29.967570 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:29.967550 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 00:05:30.012758 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.012736 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95txv\" (UniqueName: \"kubernetes.io/projected/86519442-21ae-43dc-98bc-730f8212e34b-kube-api-access-95txv\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:30.012888 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.012768 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgv8\" (UniqueName: \"kubernetes.io/projected/b962e9e5-781e-450c-8c3f-17e217c67df6-kube-api-access-sqgv8\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:30.012888 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.012787 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b962e9e5-781e-450c-8c3f-17e217c67df6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:30.012888 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.012801 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5bx\" (UniqueName: \"kubernetes.io/projected/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0-kube-api-access-gp5bx\") pod \"ingress-canary-cdsd8\" (UID: \"6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0\") " pod="openshift-ingress-canary/ingress-canary-cdsd8" Nov 03 00:05:30.012888 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.012839 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b962e9e5-781e-450c-8c3f-17e217c67df6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:30.012888 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.012861 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0-cert\") pod \"ingress-canary-cdsd8\" (UID: \"6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0\") " pod="openshift-ingress-canary/ingress-canary-cdsd8" Nov 03 00:05:30.012888 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.012886 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b962e9e5-781e-450c-8c3f-17e217c67df6-crio-socket\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:30.013158 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.012921 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86519442-21ae-43dc-98bc-730f8212e34b-config-volume\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:30.013158 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.012941 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86519442-21ae-43dc-98bc-730f8212e34b-metrics-tls\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:30.013158 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.012963 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86519442-21ae-43dc-98bc-730f8212e34b-tmp-dir\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:30.013158 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.013006 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b962e9e5-781e-450c-8c3f-17e217c67df6-data-volume\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:30.013318 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.013172 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b962e9e5-781e-450c-8c3f-17e217c67df6-crio-socket\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:30.013368 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.013337 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b962e9e5-781e-450c-8c3f-17e217c67df6-data-volume\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:30.013625 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.013606 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86519442-21ae-43dc-98bc-730f8212e34b-tmp-dir\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:30.013759 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.013708 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b962e9e5-781e-450c-8c3f-17e217c67df6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:30.013947 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.013920 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86519442-21ae-43dc-98bc-730f8212e34b-config-volume\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:30.016689 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.016668 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86519442-21ae-43dc-98bc-730f8212e34b-metrics-tls\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:30.016795 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.016729 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b962e9e5-781e-450c-8c3f-17e217c67df6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:30.016795 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.016744 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0-cert\") pod \"ingress-canary-cdsd8\" (UID: \"6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0\") " pod="openshift-ingress-canary/ingress-canary-cdsd8" Nov 03 00:05:30.070212 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.070155 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95txv\" (UniqueName: \"kubernetes.io/projected/86519442-21ae-43dc-98bc-730f8212e34b-kube-api-access-95txv\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:30.070303 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.070223 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgv8\" (UniqueName: \"kubernetes.io/projected/b962e9e5-781e-450c-8c3f-17e217c67df6-kube-api-access-sqgv8\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:30.070371 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.070355 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp5bx\" (UniqueName: \"kubernetes.io/projected/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0-kube-api-access-gp5bx\") pod \"ingress-canary-cdsd8\" (UID: \"6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0\") " pod="openshift-ingress-canary/ingress-canary-cdsd8" Nov 03 00:05:30.127237 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.127211 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:30.131389 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.131369 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cdsd8" Nov 03 00:05:30.138970 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.138952 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:30.514449 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.514369 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cdsd8"] Nov 03 00:05:30.517790 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:30.517763 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce1f854_bc96_4ed4_a93a_f7c0092bc4c0.slice/crio-044a2e3307e37f67d0309483ddf7e14c808a18edae993e284963bbc7d33fcabf WatchSource:0}: Error finding container 044a2e3307e37f67d0309483ddf7e14c808a18edae993e284963bbc7d33fcabf: Status 404 returned error can't find the container with id 044a2e3307e37f67d0309483ddf7e14c808a18edae993e284963bbc7d33fcabf Nov 03 00:05:30.560362 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.560337 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6v6bq"] Nov 03 00:05:30.563051 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:30.563027 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb962e9e5_781e_450c_8c3f_17e217c67df6.slice/crio-73b46b35c4a79d74c453a6b455511d4c957f31a7f0b1aa3f20f7c37b7c2fc344 WatchSource:0}: Error finding container 73b46b35c4a79d74c453a6b455511d4c957f31a7f0b1aa3f20f7c37b7c2fc344: Status 404 returned error can't find the container with id 73b46b35c4a79d74c453a6b455511d4c957f31a7f0b1aa3f20f7c37b7c2fc344 Nov 03 00:05:30.563133 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.563067 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rwmxk"] Nov 03 00:05:30.565457 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:30.565437 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86519442_21ae_43dc_98bc_730f8212e34b.slice/crio-590267b131f92a1931738683b05c26f3488548e2948e1cfd6418d61d71bce1c6 WatchSource:0}: Error finding container 590267b131f92a1931738683b05c26f3488548e2948e1cfd6418d61d71bce1c6: Status 404 returned error can't find the container with id 590267b131f92a1931738683b05c26f3488548e2948e1cfd6418d61d71bce1c6 Nov 03 00:05:30.754802 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.754771 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:30.754802 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.754803 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:30.755021 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.754878 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:30.807548 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.807528 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 00:05:30.807671 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.807611 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gd4nl\"" Nov 03 00:05:30.807671 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.807615 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 00:05:30.807671 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.807634 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 00:05:30.807671 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.807651 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 00:05:30.807671 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.807653 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vqr8l\"" Nov 03 00:05:30.807671 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.807670 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fmpf5\"" Nov 03 00:05:30.908684 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.908662 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:30.984763 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.984737 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rwmxk" event={"ID":"86519442-21ae-43dc-98bc-730f8212e34b","Type":"ContainerStarted","Data":"590267b131f92a1931738683b05c26f3488548e2948e1cfd6418d61d71bce1c6"} Nov 03 00:05:30.986070 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.986052 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6v6bq" event={"ID":"b962e9e5-781e-450c-8c3f-17e217c67df6","Type":"ContainerStarted","Data":"a35a750338d08ceb162d6407f0cc20018c612b0555366ecc61a8aee0516af800"} Nov 03 00:05:30.986167 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.986092 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6v6bq" event={"ID":"b962e9e5-781e-450c-8c3f-17e217c67df6","Type":"ContainerStarted","Data":"73b46b35c4a79d74c453a6b455511d4c957f31a7f0b1aa3f20f7c37b7c2fc344"} Nov 03 00:05:30.987156 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:30.987135 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cdsd8" event={"ID":"6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0","Type":"ContainerStarted","Data":"044a2e3307e37f67d0309483ddf7e14c808a18edae993e284963bbc7d33fcabf"} Nov 03 00:05:31.991924 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:31.991841 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6v6bq" event={"ID":"b962e9e5-781e-450c-8c3f-17e217c67df6","Type":"ContainerStarted","Data":"347036eeae480ffcb917d2b969297536e528b1b7c45982db0364d08ea2bfceb2"} Nov 03 00:05:32.380115 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:32.380085 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8cpwh_ddae2ef2-bac4-44c3-bef2-0d40c705c141/dns-node-resolver/0.log" Nov 03 00:05:32.997460 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:32.997380 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rwmxk" event={"ID":"86519442-21ae-43dc-98bc-730f8212e34b","Type":"ContainerStarted","Data":"132726d40c634bbaa8d1657cd8a0a45610747ed83f4b7ddfdd042ff94629a806"} Nov 03 00:05:32.997460 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:32.997447 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rwmxk" event={"ID":"86519442-21ae-43dc-98bc-730f8212e34b","Type":"ContainerStarted","Data":"b8262f36533fcf356449b77d88775a5c2cac323a714b49f5257a3b21d2441898"} Nov 03 00:05:32.998009 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:32.997533 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:33.114471 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:33.114324 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rwmxk" podStartSLOduration=2.020073836 podStartE2EDuration="4.114308123s" totalImagesPullingTime="2.094234287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:33.113929406 +0000 UTC m=+53.816533294" watchObservedRunningTime="2026-11-03 00:05:33.114308123 +0000 UTC m=+53.816912013" Nov 03 00:05:33.779444 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:33.779419 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nckb2_4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc/node-ca/0.log" Nov 03 00:05:34.001682 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.001645 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6v6bq" event={"ID":"b962e9e5-781e-450c-8c3f-17e217c67df6","Type":"ContainerStarted","Data":"96e0e5a9bbcf8b87eae9d93952b9ca362bd00b21120ec2513fd75e70bd2ca2cd"} Nov 03 00:05:34.003185 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.003155 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cdsd8" event={"ID":"6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0","Type":"ContainerStarted","Data":"51d4f4a462f39182ab1b43358747ef6d6bc381dda396d4d3bc9079c08b8b07b9"} Nov 03 00:05:34.116860 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.116774 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6v6bq" podStartSLOduration=2.043277697 podStartE2EDuration="5.116757105s" totalImagesPullingTime="3.073479408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:05:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:34.116433467 +0000 UTC m=+54.819037367" watchObservedRunningTime="2026-11-03 00:05:34.116757105 +0000 UTC m=+54.819360995" Nov 03 00:05:34.163631 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.163591 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:34.230036 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.229989 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cdsd8" podStartSLOduration=1.9863366839999999 podStartE2EDuration="5.229977968s" totalImagesPullingTime="3.243641284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:34.229441634 +0000 UTC m=+54.932045541" watchObservedRunningTime="2026-11-03 00:05:34.229977968 +0000 UTC m=+54.932581858" Nov 03 00:05:34.326320 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.326295 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:34.379585 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.379531 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cdsd8_6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0/serve-healthcheck-canary/0.log" Nov 03 00:05:34.484532 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.484507 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:34.550325 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.550301 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9"] Nov 03 00:05:34.553242 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.553219 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:34.609190 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.609155 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 00:05:34.609313 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.609258 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 00:05:34.639194 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.639131 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:34.659116 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.659085 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 00:05:34.659116 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.659089 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 00:05:34.659261 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.659174 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-znq56\"" Nov 03 00:05:34.659261 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.659235 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 00:05:34.659839 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.659805 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9"] Nov 03 00:05:34.748382 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.748361 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/169c66f9-91af-4e46-bf67-019e0cec4a43-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:34.748487 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.748407 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/169c66f9-91af-4e46-bf67-019e0cec4a43-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:34.748487 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.748427 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhkl\" (UniqueName: \"kubernetes.io/projected/169c66f9-91af-4e46-bf67-019e0cec4a43-kube-api-access-kxhkl\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:34.748567 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.748510 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/169c66f9-91af-4e46-bf67-019e0cec4a43-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:34.849277 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.849257 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/169c66f9-91af-4e46-bf67-019e0cec4a43-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:34.849392 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.849285 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhkl\" (UniqueName: \"kubernetes.io/projected/169c66f9-91af-4e46-bf67-019e0cec4a43-kube-api-access-kxhkl\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:34.849392 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.849305 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/169c66f9-91af-4e46-bf67-019e0cec4a43-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:34.849392 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.849348 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/169c66f9-91af-4e46-bf67-019e0cec4a43-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:34.850058 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.850037 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/169c66f9-91af-4e46-bf67-019e0cec4a43-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:34.851744 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.851715 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/169c66f9-91af-4e46-bf67-019e0cec4a43-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:34.851839 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.851797 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/169c66f9-91af-4e46-bf67-019e0cec4a43-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:34.905913 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:34.905859 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhkl\" (UniqueName: \"kubernetes.io/projected/169c66f9-91af-4e46-bf67-019e0cec4a43-kube-api-access-kxhkl\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:35.164325 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:35.164265 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:35.530293 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:35.530229 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9"] Nov 03 00:05:35.564880 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:35.564854 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169c66f9_91af_4e46_bf67_019e0cec4a43.slice/crio-0ad6c1d37cce5458301f9e4f1e8ebb0ceee5b80eb004af22325bf33b627cb883 WatchSource:0}: Error finding container 0ad6c1d37cce5458301f9e4f1e8ebb0ceee5b80eb004af22325bf33b627cb883: Status 404 returned error can't find the container with id 0ad6c1d37cce5458301f9e4f1e8ebb0ceee5b80eb004af22325bf33b627cb883 Nov 03 00:05:36.009366 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.009338 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" event={"ID":"169c66f9-91af-4e46-bf67-019e0cec4a43","Type":"ContainerStarted","Data":"0ad6c1d37cce5458301f9e4f1e8ebb0ceee5b80eb004af22325bf33b627cb883"} Nov 03 00:05:36.152989 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.152955 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c6c8d65bd-fhjhr"] Nov 03 00:05:36.155246 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.155225 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.257557 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.257517 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 00:05:36.257557 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.257528 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 00:05:36.257557 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.257528 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p8dw8\"" Nov 03 00:05:36.257964 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.257568 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 00:05:36.257964 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.257577 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 00:05:36.257964 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.257584 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 00:05:36.257964 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.257605 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 00:05:36.257964 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.257606 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 00:05:36.258900 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.258884 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-serving-cert\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.258947 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.258917 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-service-ca\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.258947 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.258931 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/423516a7-579e-492a-93dc-0c21e3faa673-tmp\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.259039 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.258950 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-console-config\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.259039 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.259024 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g9zg\" (UniqueName: \"kubernetes.io/projected/423516a7-579e-492a-93dc-0c21e3faa673-kube-api-access-9g9zg\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.259174 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.259137 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-trusted-ca-bundle\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.259174 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.259158 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-oauth-serving-cert\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.259255 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.259174 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-oauth-config\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.358199 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.358172 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 00:05:36.359138 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.359122 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c6c8d65bd-fhjhr"] Nov 03 00:05:36.359495 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.359477 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-serving-cert\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.359542 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.359530 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-service-ca\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.359575 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.359554 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/423516a7-579e-492a-93dc-0c21e3faa673-tmp\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.359606 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.359581 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-console-config\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.359641 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.359615 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9zg\" (UniqueName: \"kubernetes.io/projected/423516a7-579e-492a-93dc-0c21e3faa673-kube-api-access-9g9zg\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.359688 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.359641 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-trusted-ca-bundle\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.359688 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.359663 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-oauth-serving-cert\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.359768 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.359690 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-oauth-config\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.360697 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.360667 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/423516a7-579e-492a-93dc-0c21e3faa673-tmp\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.360982 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.360964 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-console-config\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.361130 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.361110 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-trusted-ca-bundle\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.361192 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.361132 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-oauth-serving-cert\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.361192 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.361146 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-service-ca\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.362288 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.362266 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-oauth-config\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.362361 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.362349 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-serving-cert\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.416012 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.415993 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9zg\" (UniqueName: \"kubernetes.io/projected/423516a7-579e-492a-93dc-0c21e3faa673-kube-api-access-9g9zg\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.465040 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.465011 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:36.668688 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.668612 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:36.830318 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.830287 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c6c8d65bd-fhjhr"] Nov 03 00:05:36.853708 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:36.853677 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:36.864346 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:36.864313 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423516a7_579e_492a_93dc_0c21e3faa673.slice/crio-806034f7a24d9e9dd18667d2c50a1a57131ab581d5b4d0bb53b4b3629a468fc7 WatchSource:0}: Error finding container 806034f7a24d9e9dd18667d2c50a1a57131ab581d5b4d0bb53b4b3629a468fc7: Status 404 returned error can't find the container with id 806034f7a24d9e9dd18667d2c50a1a57131ab581d5b4d0bb53b4b3629a468fc7 Nov 03 00:05:37.014223 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:37.014141 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6c8d65bd-fhjhr" event={"ID":"423516a7-579e-492a-93dc-0c21e3faa673","Type":"ContainerStarted","Data":"806034f7a24d9e9dd18667d2c50a1a57131ab581d5b4d0bb53b4b3629a468fc7"} Nov 03 00:05:37.948482 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:37.948440 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:38.019812 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:38.019773 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" event={"ID":"169c66f9-91af-4e46-bf67-019e0cec4a43","Type":"ContainerStarted","Data":"13c6b264ce9ca72a1b1cb62db58c9e504ab309d44eb3178d046ee15a7213a5e4"} Nov 03 00:05:38.020002 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:38.019850 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" event={"ID":"169c66f9-91af-4e46-bf67-019e0cec4a43","Type":"ContainerStarted","Data":"5e321e4e5f77ae6cc9a4ca75201c4683910d0bdec4a9350833b919cc33ffc7a4"} Nov 03 00:05:38.235378 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:38.235218 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" podStartSLOduration=2.558508452 podStartE2EDuration="4.23519946s" totalImagesPullingTime="1.676691008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:38.234499701 +0000 UTC m=+58.937103610" watchObservedRunningTime="2026-11-03 00:05:38.23519946 +0000 UTC m=+58.937803351" Nov 03 00:05:39.870622 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:39.870574 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:40.497966 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:40.497889 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:40.685793 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:40.685753 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:40.945554 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:40.945519 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:41.030988 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.030952 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6c8d65bd-fhjhr" event={"ID":"423516a7-579e-492a-93dc-0c21e3faa673","Type":"ContainerStarted","Data":"c8efae2f0a737e313a0f05a730381acee05cb70f622b64db4bd098f32871d9d8"} Nov 03 00:05:41.120309 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.120267 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:41.136719 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.136695 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk"] Nov 03 00:05:41.138623 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.138608 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.187104 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.187057 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c6c8d65bd-fhjhr" podStartSLOduration=1.9545772810000002 podStartE2EDuration="5.187042058s" totalImagesPullingTime="3.232464777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:41.149349949 +0000 UTC m=+61.851953856" watchObservedRunningTime="2026-11-03 00:05:41.187042058 +0000 UTC m=+61.889645947" Nov 03 00:05:41.188159 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.188143 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk"] Nov 03 00:05:41.194477 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.194459 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9mww\" (UniqueName: \"kubernetes.io/projected/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-kube-api-access-c9mww\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.194534 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.194492 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.194534 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.194515 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.194601 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.194563 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.199738 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.199685 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 00:05:41.199738 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.199727 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 00:05:41.199903 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.199762 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8pmbc\"" Nov 03 00:05:41.211968 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.211943 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mzcwb"] Nov 03 00:05:41.213773 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.213760 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.240632 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.240605 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s"] Nov 03 00:05:41.242926 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.242912 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.273079 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.273047 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 00:05:41.291013 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.290981 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:41.294202 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.294187 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 00:05:41.294270 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.294192 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 00:05:41.294430 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.294417 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-d8jlp\"" Nov 03 00:05:41.295111 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295086 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.295183 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295131 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 00:05:41.295183 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295144 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.295260 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295205 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5h6xs\"" Nov 03 00:05:41.295260 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295237 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9mww\" (UniqueName: \"kubernetes.io/projected/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-kube-api-access-c9mww\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.295340 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295268 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-sys\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.295340 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295275 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 00:05:41.295340 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295294 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-wtmp\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.295457 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295343 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.295457 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295360 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.295457 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295377 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.295457 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295417 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ba97f37-1485-4215-abe3-3c9083759d8a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.295607 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295483 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgv7z\" (UniqueName: \"kubernetes.io/projected/9967e995-dc16-4869-bb2f-02eb340eba3e-kube-api-access-xgv7z\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.295607 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295516 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8664\" (UniqueName: \"kubernetes.io/projected/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-api-access-l8664\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.295607 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295603 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-tls\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.295698 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295620 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.295698 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295681 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.295767 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295703 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-textfile\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.295767 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295728 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.295767 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295753 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9967e995-dc16-4869-bb2f-02eb340eba3e-metrics-client-ca\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.295927 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295772 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-root\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.295927 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.295812 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ba97f37-1485-4215-abe3-3c9083759d8a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.296134 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.296117 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.297859 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.297811 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.298023 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.298004 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.303146 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.303116 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s"] Nov 03 00:05:41.352039 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.352017 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9mww\" (UniqueName: \"kubernetes.io/projected/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-kube-api-access-c9mww\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.370644 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.370625 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 00:05:41.396316 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396292 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-tls\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.396406 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396319 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.396406 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396341 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-textfile\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.396406 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396359 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.396541 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396405 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9967e995-dc16-4869-bb2f-02eb340eba3e-metrics-client-ca\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.396541 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396434 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-root\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.396541 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396461 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ba97f37-1485-4215-abe3-3c9083759d8a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.396541 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396496 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.396707 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396541 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.396707 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396546 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-root\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.396707 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396599 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-sys\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.396707 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396627 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-wtmp\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.396707 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396659 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.397031 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396752 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ba97f37-1485-4215-abe3-3c9083759d8a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.397031 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396752 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-textfile\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.397031 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396783 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgv7z\" (UniqueName: \"kubernetes.io/projected/9967e995-dc16-4869-bb2f-02eb340eba3e-kube-api-access-xgv7z\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.397031 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396808 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8664\" (UniqueName: \"kubernetes.io/projected/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-api-access-l8664\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.397031 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.396879 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ba97f37-1485-4215-abe3-3c9083759d8a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.397282 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.397138 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9967e995-dc16-4869-bb2f-02eb340eba3e-metrics-client-ca\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.397282 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.397244 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.397347 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.397303 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-sys\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.397436 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.397419 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-wtmp\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.397699 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.397673 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.397857 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.397731 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ba97f37-1485-4215-abe3-3c9083759d8a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.398976 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.398960 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-tls\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.399169 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.399152 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.399547 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.399532 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.399689 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.399671 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.450548 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.450480 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:41.456649 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.456618 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8664\" (UniqueName: \"kubernetes.io/projected/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-api-access-l8664\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.457220 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.457196 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgv7z\" (UniqueName: \"kubernetes.io/projected/9967e995-dc16-4869-bb2f-02eb340eba3e-kube-api-access-xgv7z\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.471616 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.471582 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:41.526222 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.526186 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:41.537172 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:41.537137 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9967e995_dc16_4869_bb2f_02eb340eba3e.slice/crio-a3ce8bbd2363f3c2df2c0e5010f7fd5fae48adc55f438b6c2053fd078d7997d6 WatchSource:0}: Error finding container a3ce8bbd2363f3c2df2c0e5010f7fd5fae48adc55f438b6c2053fd078d7997d6: Status 404 returned error can't find the container with id a3ce8bbd2363f3c2df2c0e5010f7fd5fae48adc55f438b6c2053fd078d7997d6 Nov 03 00:05:41.554660 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.554635 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:41.631101 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.631072 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:41.801762 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.801739 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:41.864326 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.864295 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk"] Nov 03 00:05:41.909117 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:41.909095 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b0b5df_4723_4f99_98a0_b7a386f3d0c5.slice/crio-f84132e38423833f69d0b21d3411745548cdca2b397efb770c472bfb053d06cf WatchSource:0}: Error finding container f84132e38423833f69d0b21d3411745548cdca2b397efb770c472bfb053d06cf: Status 404 returned error can't find the container with id f84132e38423833f69d0b21d3411745548cdca2b397efb770c472bfb053d06cf Nov 03 00:05:41.919652 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:41.919630 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s"] Nov 03 00:05:41.922434 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:41.922411 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba97f37_1485_4215_abe3_3c9083759d8a.slice/crio-f6740cdf7088b72d34df1b4c08503fb08155e66c03b920e7dff823589e1863d1 WatchSource:0}: Error finding container f6740cdf7088b72d34df1b4c08503fb08155e66c03b920e7dff823589e1863d1: Status 404 returned error can't find the container with id f6740cdf7088b72d34df1b4c08503fb08155e66c03b920e7dff823589e1863d1 Nov 03 00:05:42.035805 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:42.035768 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzcwb" event={"ID":"9967e995-dc16-4869-bb2f-02eb340eba3e","Type":"ContainerStarted","Data":"a3ce8bbd2363f3c2df2c0e5010f7fd5fae48adc55f438b6c2053fd078d7997d6"} Nov 03 00:05:42.037049 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:42.037021 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" event={"ID":"8ba97f37-1485-4215-abe3-3c9083759d8a","Type":"ContainerStarted","Data":"f6740cdf7088b72d34df1b4c08503fb08155e66c03b920e7dff823589e1863d1"} Nov 03 00:05:42.039624 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:42.039601 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" event={"ID":"54b0b5df-4723-4f99-98a0-b7a386f3d0c5","Type":"ContainerStarted","Data":"cdca5a776ffd793ece4720139056f612138e6c277ef022f03e836435e4298ee4"} Nov 03 00:05:42.039713 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:42.039645 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" event={"ID":"54b0b5df-4723-4f99-98a0-b7a386f3d0c5","Type":"ContainerStarted","Data":"6ce7ed1f725a15b439b4da4cb3024a0c8a4271a2c1066794add49316539d0fcb"} Nov 03 00:05:42.039713 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:42.039666 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" event={"ID":"54b0b5df-4723-4f99-98a0-b7a386f3d0c5","Type":"ContainerStarted","Data":"f84132e38423833f69d0b21d3411745548cdca2b397efb770c472bfb053d06cf"} Nov 03 00:05:42.643286 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:42.643248 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:42.802292 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:42.802258 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:43.005759 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.005727 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:43.047541 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.047503 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzcwb" event={"ID":"9967e995-dc16-4869-bb2f-02eb340eba3e","Type":"ContainerStarted","Data":"0330831cc4834e164c12da885a184fabfe851ddc493b0edc7c9f458dd1948b1e"} Nov 03 00:05:43.466560 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.466520 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:43.677415 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.677393 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:43.862795 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.862761 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h"] Nov 03 00:05:43.865255 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.865242 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:43.917681 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.917641 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 00:05:43.917892 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.917677 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 00:05:43.917892 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.917685 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 00:05:43.917892 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.917715 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 00:05:43.917892 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.917748 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ee34tp9efdls7\"" Nov 03 00:05:43.917892 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.917761 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 00:05:43.917892 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.917748 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7jrlf\"" Nov 03 00:05:43.919109 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.919090 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-grpc-tls\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:43.919184 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.919128 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:43.919249 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.919195 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:43.919249 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.919227 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbec544b-d079-48c5-9d21-0e739d7512d4-metrics-client-ca\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:43.919338 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.919297 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl4rl\" (UniqueName: \"kubernetes.io/projected/cbec544b-d079-48c5-9d21-0e739d7512d4-kube-api-access-pl4rl\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:43.919371 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.919358 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:43.919403 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.919383 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-tls\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:43.919403 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.919399 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:43.919488 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.919475 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbec544b-d079-48c5-9d21-0e739d7512d4-tmp\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:43.926269 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:43.926247 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h"] Nov 03 00:05:44.020347 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.020317 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.020501 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.020369 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbec544b-d079-48c5-9d21-0e739d7512d4-tmp\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.020501 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.020414 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-grpc-tls\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.020501 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.020444 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.020501 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.020490 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.020630 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.020543 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbec544b-d079-48c5-9d21-0e739d7512d4-metrics-client-ca\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.020630 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.020608 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl4rl\" (UniqueName: \"kubernetes.io/projected/cbec544b-d079-48c5-9d21-0e739d7512d4-kube-api-access-pl4rl\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.020694 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.020656 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.020734 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.020692 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-tls\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.020888 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.020816 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbec544b-d079-48c5-9d21-0e739d7512d4-tmp\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.021479 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.021446 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbec544b-d079-48c5-9d21-0e739d7512d4-metrics-client-ca\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.023114 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.023091 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.023321 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.023287 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.023750 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.023728 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-grpc-tls\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.023851 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.023775 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.023903 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.023868 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.023981 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.023964 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-tls\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.056968 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.055796 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" event={"ID":"8ba97f37-1485-4215-abe3-3c9083759d8a","Type":"ContainerStarted","Data":"7381f29e570586a3c9a6c45abcc2b731dc5b43233e873ff0972be6565abc8627"} Nov 03 00:05:44.056968 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.055880 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" event={"ID":"8ba97f37-1485-4215-abe3-3c9083759d8a","Type":"ContainerStarted","Data":"f4633778c32710c752ceae68f8e20cfd2f86932112f0339239462c51cbc842f8"} Nov 03 00:05:44.056968 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.055912 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" event={"ID":"8ba97f37-1485-4215-abe3-3c9083759d8a","Type":"ContainerStarted","Data":"d89b8c473fe74001d6b7c2a9d6e576d8e18287790ac84e6f556dfae042efddda"} Nov 03 00:05:44.061531 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.061500 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" event={"ID":"54b0b5df-4723-4f99-98a0-b7a386f3d0c5","Type":"ContainerStarted","Data":"f0446f3bdcfec543e42739dfc5b4e8842eb59aed6a5d3c92f14ca24a82c4cf05"} Nov 03 00:05:44.063135 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.063114 2571 generic.go:356] "Generic (PLEG): container finished" podID="9967e995-dc16-4869-bb2f-02eb340eba3e" containerID="0330831cc4834e164c12da885a184fabfe851ddc493b0edc7c9f458dd1948b1e" exitCode=0 Nov 03 00:05:44.063238 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.063147 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzcwb" event={"ID":"9967e995-dc16-4869-bb2f-02eb340eba3e","Type":"ContainerDied","Data":"0330831cc4834e164c12da885a184fabfe851ddc493b0edc7c9f458dd1948b1e"} Nov 03 00:05:44.079533 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.079509 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl4rl\" (UniqueName: \"kubernetes.io/projected/cbec544b-d079-48c5-9d21-0e739d7512d4-kube-api-access-pl4rl\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.174878 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.174806 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" podStartSLOduration=1.5913972969999999 podStartE2EDuration="3.174792611s" totalImagesPullingTime="1.583395314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:44.174247981 +0000 UTC m=+64.876851888" watchObservedRunningTime="2026-11-03 00:05:44.174792611 +0000 UTC m=+64.877396497" Nov 03 00:05:44.179698 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.179676 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:44.190664 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.190623 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:44.295522 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.295471 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" podStartSLOduration=1.8411448849999998 podStartE2EDuration="3.295452518s" totalImagesPullingTime="1.454307633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:44.294868249 +0000 UTC m=+64.997472156" watchObservedRunningTime="2026-11-03 00:05:44.295452518 +0000 UTC m=+64.998056407" Nov 03 00:05:44.390983 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.390905 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:44.612807 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.612786 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h"] Nov 03 00:05:44.615211 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:44.615180 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbec544b_d079_48c5_9d21_0e739d7512d4.slice/crio-79a3239bf922fe4e37346fd25a982ba6b37aa13b134ae1808e370f7390ed6e74 WatchSource:0}: Error finding container 79a3239bf922fe4e37346fd25a982ba6b37aa13b134ae1808e370f7390ed6e74: Status 404 returned error can't find the container with id 79a3239bf922fe4e37346fd25a982ba6b37aa13b134ae1808e370f7390ed6e74 Nov 03 00:05:44.727242 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.727169 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:44.727242 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.727237 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:44.778600 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.778571 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 00:05:44.778704 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.778650 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 00:05:44.790090 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.790068 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:44.790190 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.790181 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:44.828091 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.828066 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrpd\" (UniqueName: \"kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd\") pod \"network-check-target-x6rmn\" (UID: \"fd761dae-da21-44e4-adad-287a5d204bd3\") " pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:44.879406 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.879384 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 00:05:44.916449 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.916426 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vqr8l\"" Nov 03 00:05:44.923232 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.923211 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fmpf5\"" Nov 03 00:05:44.925379 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.925363 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:44.933173 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.933148 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:44.940532 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.940507 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 00:05:44.952420 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:44.952396 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrpd\" (UniqueName: \"kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd\") pod \"network-check-target-x6rmn\" (UID: \"fd761dae-da21-44e4-adad-287a5d204bd3\") " pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:45.068872 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.068804 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzcwb" event={"ID":"9967e995-dc16-4869-bb2f-02eb340eba3e","Type":"ContainerStarted","Data":"eace0dfa07dfe65c1c5d3a5d1bd178c8615f0ca3ce23f9292391629b9f9ad229"} Nov 03 00:05:45.069287 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.068920 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzcwb" event={"ID":"9967e995-dc16-4869-bb2f-02eb340eba3e","Type":"ContainerStarted","Data":"c94f5f791b524c5e135b72397325ece0ebf3d2caae6014d521575eb812fef0b4"} Nov 03 00:05:45.070153 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.070131 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" event={"ID":"cbec544b-d079-48c5-9d21-0e739d7512d4","Type":"ContainerStarted","Data":"79a3239bf922fe4e37346fd25a982ba6b37aa13b134ae1808e370f7390ed6e74"} Nov 03 00:05:45.072770 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.072713 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:45.188188 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.188134 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mzcwb" podStartSLOduration=2.759846024 podStartE2EDuration="4.188115888s" totalImagesPullingTime="1.336938474s" totalInitContainerRuntime="91.33139ms" isStatefulPod=true podCreationTimestamp="2026-11-03 00:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:45.187632967 +0000 UTC m=+65.890236858" watchObservedRunningTime="2026-11-03 00:05:45.188115888 +0000 UTC m=+65.890720011" Nov 03 00:05:45.225078 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.225040 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:45.226283 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.226248 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gd4nl\"" Nov 03 00:05:45.235532 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.235468 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:45.298390 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.298363 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-52cjz"] Nov 03 00:05:45.298530 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:45.298419 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf82aaa_e340_4a97_a3f4_297694eccab3.slice/crio-a26db85a1fec77b5424925f7f25462f608ccc68b6ce178f8665b510ac870f91b WatchSource:0}: Error finding container a26db85a1fec77b5424925f7f25462f608ccc68b6ce178f8665b510ac870f91b: Status 404 returned error can't find the container with id a26db85a1fec77b5424925f7f25462f608ccc68b6ce178f8665b510ac870f91b Nov 03 00:05:45.345619 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.345598 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-67vkk"] Nov 03 00:05:45.347601 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:45.347578 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea31369b_b8c9_4405_ab0c_6796cd41cd7a.slice/crio-348a3f58262f547f6b4d5e1c1a847ac135375ac640bec29138e6575a25c8ec8e WatchSource:0}: Error finding container 348a3f58262f547f6b4d5e1c1a847ac135375ac640bec29138e6575a25c8ec8e: Status 404 returned error can't find the container with id 348a3f58262f547f6b4d5e1c1a847ac135375ac640bec29138e6575a25c8ec8e Nov 03 00:05:45.380776 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.380740 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:45.602681 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.602655 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x6rmn"] Nov 03 00:05:45.606047 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:45.606024 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd761dae_da21_44e4_adad_287a5d204bd3.slice/crio-b48cb7337658a344140015fd5fd7043c04af3900043258a64a07b9b86be94398 WatchSource:0}: Error finding container b48cb7337658a344140015fd5fd7043c04af3900043258a64a07b9b86be94398: Status 404 returned error can't find the container with id b48cb7337658a344140015fd5fd7043c04af3900043258a64a07b9b86be94398 Nov 03 00:05:45.609665 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.609639 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:45.771983 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.771947 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:45.936513 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.936436 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:45.943129 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.943104 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm"] Nov 03 00:05:45.945087 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.945056 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:45.997677 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.997658 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-sfwn4\"" Nov 03 00:05:45.997677 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:45.997665 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 00:05:46.002292 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.002086 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm"] Nov 03 00:05:46.038275 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.038251 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-745647dff6-lq2qf"] Nov 03 00:05:46.039524 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.039500 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443202b2-a46c-41af-b097-93658ee871f8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vcbjm\" (UID: \"443202b2-a46c-41af-b097-93658ee871f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:46.039634 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.039535 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6lrg\" (UniqueName: \"kubernetes.io/projected/443202b2-a46c-41af-b097-93658ee871f8-kube-api-access-j6lrg\") pod \"monitoring-plugin-77f4f76594-vcbjm\" (UID: \"443202b2-a46c-41af-b097-93658ee871f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:46.040545 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.040528 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.075324 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.075289 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x6rmn" event={"ID":"fd761dae-da21-44e4-adad-287a5d204bd3","Type":"ContainerStarted","Data":"b48cb7337658a344140015fd5fd7043c04af3900043258a64a07b9b86be94398"} Nov 03 00:05:46.076767 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.076733 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-67vkk" event={"ID":"ea31369b-b8c9-4405-ab0c-6796cd41cd7a","Type":"ContainerStarted","Data":"348a3f58262f547f6b4d5e1c1a847ac135375ac640bec29138e6575a25c8ec8e"} Nov 03 00:05:46.078075 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.078041 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-52cjz" event={"ID":"fdf82aaa-e340-4a97-a3f4-297694eccab3","Type":"ContainerStarted","Data":"a26db85a1fec77b5424925f7f25462f608ccc68b6ce178f8665b510ac870f91b"} Nov 03 00:05:46.085787 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.085765 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:46.093390 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.093364 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 00:05:46.093494 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.093475 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 00:05:46.097775 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.097673 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 00:05:46.097904 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.097888 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 00:05:46.097960 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.097927 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nxwbs\"" Nov 03 00:05:46.097960 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.097935 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bspg5up4v8aee\"" Nov 03 00:05:46.098794 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.098766 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-745647dff6-lq2qf"] Nov 03 00:05:46.140979 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.140954 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-client-ca-bundle\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.141127 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.141010 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9ee339b2-b471-4efd-829e-dd247c78685a-metrics-server-audit-profiles\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.141232 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.141212 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ee339b2-b471-4efd-829e-dd247c78685a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.141293 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.141240 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9ee339b2-b471-4efd-829e-dd247c78685a-audit-log\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.141400 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.141383 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-secret-metrics-server-tls\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.141445 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.141412 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-secret-metrics-server-client-certs\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.141479 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.141442 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7qzc\" (UniqueName: \"kubernetes.io/projected/9ee339b2-b471-4efd-829e-dd247c78685a-kube-api-access-x7qzc\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.141509 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.141475 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443202b2-a46c-41af-b097-93658ee871f8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vcbjm\" (UID: \"443202b2-a46c-41af-b097-93658ee871f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:46.141541 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.141506 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6lrg\" (UniqueName: \"kubernetes.io/projected/443202b2-a46c-41af-b097-93658ee871f8-kube-api-access-j6lrg\") pod \"monitoring-plugin-77f4f76594-vcbjm\" (UID: \"443202b2-a46c-41af-b097-93658ee871f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:46.144468 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.144441 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443202b2-a46c-41af-b097-93658ee871f8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vcbjm\" (UID: \"443202b2-a46c-41af-b097-93658ee871f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:46.199189 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.199117 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6lrg\" (UniqueName: \"kubernetes.io/projected/443202b2-a46c-41af-b097-93658ee871f8-kube-api-access-j6lrg\") pod \"monitoring-plugin-77f4f76594-vcbjm\" (UID: \"443202b2-a46c-41af-b097-93658ee871f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:46.237355 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.237304 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:46.242382 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.242353 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-secret-metrics-server-tls\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.242501 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.242397 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-secret-metrics-server-client-certs\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.242501 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.242430 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7qzc\" (UniqueName: \"kubernetes.io/projected/9ee339b2-b471-4efd-829e-dd247c78685a-kube-api-access-x7qzc\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.242501 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.242476 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-client-ca-bundle\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.242658 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.242505 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9ee339b2-b471-4efd-829e-dd247c78685a-metrics-server-audit-profiles\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.242979 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.242898 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ee339b2-b471-4efd-829e-dd247c78685a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.243074 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.242992 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9ee339b2-b471-4efd-829e-dd247c78685a-audit-log\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.243950 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.243925 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9ee339b2-b471-4efd-829e-dd247c78685a-audit-log\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.244047 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.243976 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ee339b2-b471-4efd-829e-dd247c78685a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.244103 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.244080 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9ee339b2-b471-4efd-829e-dd247c78685a-metrics-server-audit-profiles\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.245671 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.245631 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-client-ca-bundle\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.245893 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.245874 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-secret-metrics-server-tls\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.246330 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.246307 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-secret-metrics-server-client-certs\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.255385 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.255325 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:46.301128 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.301075 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7qzc\" (UniqueName: \"kubernetes.io/projected/9ee339b2-b471-4efd-829e-dd247c78685a-kube-api-access-x7qzc\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.351965 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.351629 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:46.390374 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.390325 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:46.465620 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.465557 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:46.465946 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.465923 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:46.471611 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.471585 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:46.640516 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.640495 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm"] Nov 03 00:05:46.642722 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:46.642692 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443202b2_a46c_41af_b097_93658ee871f8.slice/crio-82d46560ca8e32594be1a17cf469209e0363dc04f76c7d543a3fd843f5731b0c WatchSource:0}: Error finding container 82d46560ca8e32594be1a17cf469209e0363dc04f76c7d543a3fd843f5731b0c: Status 404 returned error can't find the container with id 82d46560ca8e32594be1a17cf469209e0363dc04f76c7d543a3fd843f5731b0c Nov 03 00:05:46.742309 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.742284 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-745647dff6-lq2qf"] Nov 03 00:05:46.921466 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.921426 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d6548ffc8-8tldc"] Nov 03 00:05:46.928808 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.928746 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:46.948778 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.948747 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-serving-certs-ca-bundle\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:46.948910 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.948791 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-federate-client-tls\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:46.948910 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.948815 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-telemeter-client-tls\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:46.948978 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.948916 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:46.949007 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.948973 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-metrics-client-ca\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:46.949057 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.949038 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:46.949092 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.949081 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-secret-telemeter-client\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:46.949134 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.949119 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztn7x\" (UniqueName: \"kubernetes.io/projected/d648f173-aa93-4fe7-999b-b50c88be3440-kube-api-access-ztn7x\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:46.982311 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.982281 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 00:05:46.982677 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:46.982656 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 00:05:47.031895 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.031793 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 00:05:47.032042 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.031911 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8kfz4\"" Nov 03 00:05:47.032155 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.032096 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 00:05:47.032155 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.032141 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 00:05:47.032950 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.032906 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 00:05:47.035148 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.035113 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d6548ffc8-8tldc"] Nov 03 00:05:47.049723 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.049665 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-serving-certs-ca-bundle\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.049723 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.049700 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-federate-client-tls\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.049908 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.049728 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-telemeter-client-tls\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.049908 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.049756 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.050865 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.050263 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-metrics-client-ca\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.050865 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.050355 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.050865 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.050393 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-secret-telemeter-client\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.050865 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.050432 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztn7x\" (UniqueName: \"kubernetes.io/projected/d648f173-aa93-4fe7-999b-b50c88be3440-kube-api-access-ztn7x\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.050865 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.050451 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-serving-certs-ca-bundle\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.051412 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.051384 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-metrics-client-ca\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.051688 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.051664 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.053756 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.053027 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.053756 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.053249 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-federate-client-tls\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.059980 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.054331 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-telemeter-client-tls\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.059980 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.055893 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-secret-telemeter-client\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.083850 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.083770 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" event={"ID":"9ee339b2-b471-4efd-829e-dd247c78685a","Type":"ContainerStarted","Data":"b1280e8938be75acaed69f833c5b3113547df5dd3eca07ce0d154d997bd8a551"} Nov 03 00:05:47.087786 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.087757 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" event={"ID":"443202b2-a46c-41af-b097-93658ee871f8","Type":"ContainerStarted","Data":"82d46560ca8e32594be1a17cf469209e0363dc04f76c7d543a3fd843f5731b0c"} Nov 03 00:05:47.093353 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.093325 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:47.108679 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.108653 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztn7x\" (UniqueName: \"kubernetes.io/projected/d648f173-aa93-4fe7-999b-b50c88be3440-kube-api-access-ztn7x\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.246897 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.246692 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:47.687533 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:47.687273 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d6548ffc8-8tldc"] Nov 03 00:05:47.692126 ip-10-0-135-199 kubenswrapper[2571]: W1103 00:05:47.692092 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd648f173_aa93_4fe7_999b_b50c88be3440.slice/crio-e8c73a3b0a3305e7ee8b22db5e28b8f2d526dfe05628c52bf487cba213a2a1fd WatchSource:0}: Error finding container e8c73a3b0a3305e7ee8b22db5e28b8f2d526dfe05628c52bf487cba213a2a1fd: Status 404 returned error can't find the container with id e8c73a3b0a3305e7ee8b22db5e28b8f2d526dfe05628c52bf487cba213a2a1fd Nov 03 00:05:48.097334 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:48.097298 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-67vkk" event={"ID":"ea31369b-b8c9-4405-ab0c-6796cd41cd7a","Type":"ContainerStarted","Data":"0202cc8eeadb3dee3564bf16421545335443adbaa1bd02faec53c5ebb4f56c93"} Nov 03 00:05:48.097734 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:48.097367 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-67vkk" event={"ID":"ea31369b-b8c9-4405-ab0c-6796cd41cd7a","Type":"ContainerStarted","Data":"8530eeb508e99c53795b1a1e0c325b4cf275a759ffafac0a63c8d70342147e36"} Nov 03 00:05:48.100295 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:48.100247 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" event={"ID":"d648f173-aa93-4fe7-999b-b50c88be3440","Type":"ContainerStarted","Data":"e8c73a3b0a3305e7ee8b22db5e28b8f2d526dfe05628c52bf487cba213a2a1fd"} Nov 03 00:05:49.112643 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:49.112603 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" event={"ID":"cbec544b-d079-48c5-9d21-0e739d7512d4","Type":"ContainerStarted","Data":"3c7800df803eff6d72ad336ced95d991ce4047db23fa6091eeb14711d662d97e"} Nov 03 00:05:49.113162 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:49.112727 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" event={"ID":"cbec544b-d079-48c5-9d21-0e739d7512d4","Type":"ContainerStarted","Data":"fec9f3e79a85d13ebb49b129049f1f78bb3e68e83a6174522a93f48a2fed783f"} Nov 03 00:05:49.113162 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:49.112781 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" event={"ID":"cbec544b-d079-48c5-9d21-0e739d7512d4","Type":"ContainerStarted","Data":"db7433edd13f97bd1496d5e5833c5d1861bcde7d2a80707e27f3502727e00b67"} Nov 03 00:05:50.123500 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:50.123455 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" event={"ID":"9ee339b2-b471-4efd-829e-dd247c78685a","Type":"ContainerStarted","Data":"2fa8df6c9971292097d37f418bf40e0e86078bbf369afe832133fb5db5b23fa1"} Nov 03 00:05:50.126692 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:50.126662 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" event={"ID":"443202b2-a46c-41af-b097-93658ee871f8","Type":"ContainerStarted","Data":"5d32c7941d04ef791a799925805ca9392f504ab6e1d7116e4dae21834c54e824"} Nov 03 00:05:50.126853 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:50.126806 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:50.132783 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:50.132739 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" event={"ID":"cbec544b-d079-48c5-9d21-0e739d7512d4","Type":"ContainerStarted","Data":"ba37e63cd04df1ba0d62e1269a6f847bc01641b063c58b0f117c5e5ecc5d97ed"} Nov 03 00:05:50.133639 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:50.133608 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" event={"ID":"cbec544b-d079-48c5-9d21-0e739d7512d4","Type":"ContainerStarted","Data":"5b96e9f40cc7ed9f66a055e434c5ec1cece0a854bef0495470b76d45e87ac400"} Nov 03 00:05:50.133742 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:50.133681 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:50.133742 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:50.133723 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" event={"ID":"cbec544b-d079-48c5-9d21-0e739d7512d4","Type":"ContainerStarted","Data":"dd4418e4043f041ab81483fbf425a299f67a394c20708c57f7df1e2194343198"} Nov 03 00:05:50.134118 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:50.134071 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:50.221675 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:50.221625 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:50.241077 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:50.240332 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" podStartSLOduration=2.690701071 podStartE2EDuration="5.240314066s" totalImagesPullingTime="2.549612995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:50.239915064 +0000 UTC m=+70.942518978" watchObservedRunningTime="2026-11-03 00:05:50.240314066 +0000 UTC m=+70.942917953" Nov 03 00:05:50.245057 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:50.244167 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-67vkk" podStartSLOduration=69.47207554 podStartE2EDuration="1m11.24414785s" totalImagesPullingTime="1.77207231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:48.218921743 +0000 UTC m=+68.921525636" watchObservedRunningTime="2026-11-03 00:05:50.24414785 +0000 UTC m=+70.946751739" Nov 03 00:05:50.354842 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:50.354515 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" podStartSLOduration=2.821156189 podStartE2EDuration="5.354499461s" totalImagesPullingTime="2.533343272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:50.35238019 +0000 UTC m=+71.054984100" watchObservedRunningTime="2026-11-03 00:05:50.354499461 +0000 UTC m=+71.057103347" Nov 03 00:05:50.422426 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:50.422267 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:51.141294 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:51.141207 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" event={"ID":"d648f173-aa93-4fe7-999b-b50c88be3440","Type":"ContainerStarted","Data":"9d9bc17d17f55f0c3ac0d43bc3bcee09d9e300419d7185aaed75de6247a4b8d7"} Nov 03 00:05:51.179723 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:51.179688 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:51.391604 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:51.391471 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:51.586344 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:51.586271 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:52.156431 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:52.156394 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x6rmn" event={"ID":"fd761dae-da21-44e4-adad-287a5d204bd3","Type":"ContainerStarted","Data":"48d7d9d016f48c08988dd446a7dbff50712641fbe823fa71361b61d79216237b"} Nov 03 00:05:52.156918 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:52.156653 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:52.271860 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:52.271783 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" podStartSLOduration=4.4273480880000005 podStartE2EDuration="9.271763322s" totalImagesPullingTime="4.844415234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:50.478265228 +0000 UTC m=+71.180869120" watchObservedRunningTime="2026-11-03 00:05:52.271763322 +0000 UTC m=+72.974367213" Nov 03 00:05:52.272257 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:52.272227 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x6rmn" podStartSLOduration=67.721908465 podStartE2EDuration="1m13.272215753s" totalImagesPullingTime="5.550307288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:52.271470489 +0000 UTC m=+72.974074389" watchObservedRunningTime="2026-11-03 00:05:52.272215753 +0000 UTC m=+72.974819647" Nov 03 00:05:52.301130 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:52.301043 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:52.480021 ip-10-0-135-199 kubenswrapper[2571]: I1103 00:05:52.479964 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:52.974838 ip-10-0-135-199 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 00:05:52.991301 ip-10-0-135-199 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 00:05:52.991471 ip-10-0-135-199 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 00:05:52.991519 ip-10-0-135-199 systemd[1]: kubelet.service: Consumed 4.312s CPU time. Nov 03 00:05:53.001321 ip-10-0-135-199 systemd[1]: Starting Kubernetes Kubelet... Nov 03 00:05:53.085462 ip-10-0-135-199 kubenswrapper[6400]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:05:53.085462 ip-10-0-135-199 kubenswrapper[6400]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 00:05:53.085462 ip-10-0-135-199 kubenswrapper[6400]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:05:53.085462 ip-10-0-135-199 kubenswrapper[6400]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:05:53.087435 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087320 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:53.087435 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087435 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:53.087435 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087441 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087447 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087452 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087465 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087470 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087474 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087478 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087483 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087487 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087491 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087496 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087500 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087504 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087509 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087513 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087517 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087521 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087525 6400 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087530 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087534 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:53.087596 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087538 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087542 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087547 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087551 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087555 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087560 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087564 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087568 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087572 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087576 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087581 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087585 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087590 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087594 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087598 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087602 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087607 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087611 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087616 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087620 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:53.088099 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087624 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087628 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087632 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087637 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087641 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087647 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087651 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087659 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087665 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087670 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087674 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087678 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087682 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087686 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087691 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087695 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087699 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087703 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087708 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:53.088581 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087712 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087716 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087720 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087724 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087729 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087733 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087738 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087742 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087746 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087750 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087755 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087759 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087763 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087767 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087771 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087775 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087779 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087783 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087787 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087792 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:53.089070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087797 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087803 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087810 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087815 6400 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087836 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087841 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087845 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087849 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087853 6400 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087857 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087863 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087867 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087872 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087875 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087880 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087884 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087887 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087892 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087896 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:53.089550 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087900 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087904 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087908 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087912 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087917 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.087922 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089191 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089202 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089207 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089213 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089217 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089221 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089226 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089230 6400 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089234 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089239 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089244 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089248 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089253 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:53.090025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089257 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089261 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089265 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089269 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089274 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089277 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089282 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089285 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089289 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089293 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089297 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089301 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089305 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089309 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089313 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089317 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089321 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089325 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089330 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:53.090495 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089334 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089338 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089342 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089346 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089351 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089355 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089359 6400 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089365 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089371 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089377 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089381 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089385 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089389 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089393 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089397 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089401 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089405 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089409 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089413 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089417 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:53.091025 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089421 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089425 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089429 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089433 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089437 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089441 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089445 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089449 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089453 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089457 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089461 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089465 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089470 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089474 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089479 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089483 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089487 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089491 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089497 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089504 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:53.091510 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089508 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089512 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089516 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089520 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089524 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089528 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089532 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089536 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089541 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089545 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089549 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089554 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089558 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089562 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089566 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089570 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089573 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089577 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089581 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:53.092052 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089585 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089589 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089594 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089598 6400 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089624 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089631 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089637 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089642 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089647 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089651 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089655 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089659 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089663 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089667 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.089671 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090482 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090491 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090496 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090501 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090505 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:53.092772 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090510 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090514 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090518 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090522 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090526 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090531 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090535 6400 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090540 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090544 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090548 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090552 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090556 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090561 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090565 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090569 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090573 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090577 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090582 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090586 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090590 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:53.093554 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090594 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090599 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090603 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090607 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090612 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090616 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090620 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090624 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090629 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090633 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090637 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090641 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090646 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090650 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090654 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090658 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090662 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090666 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090671 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090675 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:53.094288 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090680 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090684 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090688 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090695 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090702 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090709 6400 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090713 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090717 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090721 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090725 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090729 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090733 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090738 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090742 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090746 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090750 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090754 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090759 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090763 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:53.094781 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090767 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090771 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090775 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090779 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090783 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090787 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090791 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090795 6400 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090803 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090807 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090810 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090814 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090838 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090844 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090852 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090857 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090862 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090866 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090870 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:53.095256 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090874 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090878 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090882 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090886 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090889 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090893 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090898 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090902 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090906 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090910 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090914 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090920 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090924 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090928 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090932 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090936 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090940 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090944 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090948 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:53.095705 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090952 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090956 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090961 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.090965 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091099 6400 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091109 6400 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091120 6400 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091127 6400 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091136 6400 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091141 6400 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091148 6400 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091154 6400 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091159 6400 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091163 6400 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091169 6400 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091174 6400 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091179 6400 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091184 6400 flags.go:64] FLAG: --cgroup-root="" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091188 6400 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091193 6400 flags.go:64] FLAG: --client-ca-file="" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091198 6400 flags.go:64] FLAG: --cloud-provider="external" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091203 6400 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091209 6400 flags.go:64] FLAG: --cluster-domain="" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091213 6400 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 00:05:53.096215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091218 6400 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091223 6400 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091228 6400 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091234 6400 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091238 6400 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091243 6400 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091249 6400 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091253 6400 flags.go:64] FLAG: --contention-profiling="false" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091257 6400 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091262 6400 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091268 6400 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091273 6400 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091280 6400 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091284 6400 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091289 6400 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091293 6400 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091300 6400 flags.go:64] FLAG: --enable-server="true" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091304 6400 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091311 6400 flags.go:64] FLAG: --event-burst="100" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091316 6400 flags.go:64] FLAG: --event-qps="50" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091320 6400 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091325 6400 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091330 6400 flags.go:64] FLAG: --eviction-hard="" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091336 6400 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 00:05:53.097051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091340 6400 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091345 6400 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091349 6400 flags.go:64] FLAG: --eviction-soft="" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091354 6400 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091359 6400 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091364 6400 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091369 6400 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091374 6400 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091378 6400 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091383 6400 flags.go:64] FLAG: --feature-gates="" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091389 6400 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091394 6400 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091398 6400 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091403 6400 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091408 6400 flags.go:64] FLAG: --healthz-port="10248" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091413 6400 flags.go:64] FLAG: --help="false" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091417 6400 flags.go:64] FLAG: --hostname-override="ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091422 6400 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091428 6400 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091433 6400 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091438 6400 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091443 6400 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091448 6400 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 00:05:53.098014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091452 6400 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091458 6400 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091463 6400 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091468 6400 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091473 6400 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091477 6400 flags.go:64] FLAG: --kube-reserved="" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091482 6400 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091486 6400 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091491 6400 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091495 6400 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091500 6400 flags.go:64] FLAG: --lock-file="" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091506 6400 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091511 6400 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091516 6400 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091524 6400 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091530 6400 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091534 6400 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091539 6400 flags.go:64] FLAG: --logging-format="text" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091543 6400 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091549 6400 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091553 6400 flags.go:64] FLAG: --manifest-url="" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091558 6400 flags.go:64] FLAG: --manifest-url-header="" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091565 6400 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091570 6400 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091576 6400 flags.go:64] FLAG: --max-pods="110" Nov 03 00:05:53.098628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091580 6400 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091585 6400 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091591 6400 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091595 6400 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091600 6400 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091605 6400 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091609 6400 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091620 6400 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091626 6400 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091631 6400 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091637 6400 flags.go:64] FLAG: --pod-cidr="" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091641 6400 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091645 6400 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091650 6400 flags.go:64] FLAG: --pods-per-core="0" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091655 6400 flags.go:64] FLAG: --port="10250" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091659 6400 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091664 6400 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0ffbcdb5cf358393a" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091669 6400 flags.go:64] FLAG: --qos-reserved="" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091673 6400 flags.go:64] FLAG: --read-only-port="10255" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091678 6400 flags.go:64] FLAG: --register-node="true" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091682 6400 flags.go:64] FLAG: --register-with-taints="" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091689 6400 flags.go:64] FLAG: --registry-burst="10" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091693 6400 flags.go:64] FLAG: --registry-qps="5" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091698 6400 flags.go:64] FLAG: --reserved-cpus="" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091703 6400 flags.go:64] FLAG: --reserved-memory="" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091708 6400 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 00:05:53.099243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091713 6400 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091718 6400 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091722 6400 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091727 6400 flags.go:64] FLAG: --runonce="false" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091731 6400 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091736 6400 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091741 6400 flags.go:64] FLAG: --seccomp-default="false" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091745 6400 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091751 6400 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091756 6400 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091761 6400 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091765 6400 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091769 6400 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091774 6400 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091779 6400 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091783 6400 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091788 6400 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091793 6400 flags.go:64] FLAG: --system-cgroups="" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091798 6400 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091806 6400 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091810 6400 flags.go:64] FLAG: --tls-cert-file="" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091815 6400 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091840 6400 flags.go:64] FLAG: --tls-min-version="" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091844 6400 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091848 6400 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 00:05:53.099865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091852 6400 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091856 6400 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091859 6400 flags.go:64] FLAG: --v="2" Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091870 6400 flags.go:64] FLAG: --version="false" Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091875 6400 flags.go:64] FLAG: --vmodule="" Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091881 6400 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.091885 6400 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092094 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092100 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092106 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092110 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092113 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092120 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092127 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092132 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092138 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092143 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092147 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092151 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092155 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:53.100670 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092159 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092163 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092167 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092171 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092175 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092179 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092184 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092188 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092192 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092196 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092200 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092204 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092207 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092211 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092215 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092219 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092223 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092227 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092232 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092236 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:53.101432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092240 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092244 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092248 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092252 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092257 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092261 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092265 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092270 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092274 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092278 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092282 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092286 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092290 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092294 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092298 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092302 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092306 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092310 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092315 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:53.102192 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092319 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092323 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092327 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092332 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092336 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092339 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092343 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092348 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092351 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092356 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092359 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092363 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092367 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092372 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092375 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092379 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092383 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092387 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092393 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092396 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:53.102910 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092402 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092407 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092411 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092414 6400 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092418 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092422 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092425 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092429 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092433 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092437 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092440 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092444 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092448 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092452 6400 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092456 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092460 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092464 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092468 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092473 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092478 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:53.103400 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092483 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092488 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092492 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092496 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092500 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092504 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092508 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092512 6400 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092515 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092520 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092525 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092529 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092535 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092539 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.092584 6400 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:05:53.103964 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092637 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092644 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092650 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092656 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092661 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092665 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092669 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092673 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092677 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092681 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092685 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092689 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092693 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092697 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092701 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092705 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092709 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092713 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092717 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:53.104333 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092721 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092725 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092729 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092732 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092737 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092741 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092745 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092749 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092755 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092759 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092763 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092769 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092773 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092777 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092781 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092793 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092797 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092801 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092805 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092809 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:53.104864 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092813 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092835 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092839 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092843 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092847 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092851 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092855 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092859 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092862 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092866 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092870 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092874 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092878 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092882 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092886 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092890 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092894 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092899 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092903 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:53.105374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092906 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092912 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092916 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092920 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092925 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092929 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092933 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092937 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092941 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092947 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092951 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092955 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092959 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092963 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092967 6400 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092971 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092974 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092978 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092982 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092986 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:53.105895 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092990 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092994 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.092998 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093001 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093006 6400 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093010 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093013 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093017 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093021 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093027 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093032 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093037 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093042 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093047 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093052 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093056 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093060 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093064 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093068 6400 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:53.106432 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093072 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093077 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093081 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093084 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093088 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093092 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093097 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093101 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.093106 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.098751 6400 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.098768 6400 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.098784 6400 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.098788 6400 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098860 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098865 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098868 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098871 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098875 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098879 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:53.106965 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098883 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098885 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098888 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098891 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098893 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098896 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098898 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098901 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098904 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098906 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098909 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098912 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098915 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098917 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098920 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098923 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098925 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098927 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098930 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098933 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:53.107484 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098935 6400 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098937 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098940 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098944 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098946 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098949 6400 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098951 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098953 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098956 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098959 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098961 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098964 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098967 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098970 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098972 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098975 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098977 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098979 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098982 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098984 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:53.108045 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098993 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098996 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.098998 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099001 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099003 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099006 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099008 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099010 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099013 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099015 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099018 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099020 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099022 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099025 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099027 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099029 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099032 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099035 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099037 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099039 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:53.108531 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099042 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099044 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099046 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099049 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099051 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099054 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099056 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099059 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099061 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099063 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099066 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099068 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099071 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099074 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099076 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099079 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099081 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099084 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099086 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:53.109028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099089 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099091 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099094 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099096 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099099 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099101 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099105 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099108 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099110 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099113 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099115 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099118 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099120 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099123 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099125 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099127 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099130 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099132 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099135 6400 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099137 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:53.109527 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099139 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.099167 6400 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099302 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099306 6400 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099309 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099312 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099314 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099318 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099320 6400 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099323 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099325 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099327 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099330 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099332 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099334 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099337 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:53.110015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099339 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099341 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099345 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099347 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099350 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099352 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099355 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099357 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099360 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099362 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099364 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099367 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099369 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099371 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099374 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099376 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099379 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099381 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099383 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099386 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:53.110383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099388 6400 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099390 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099393 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099395 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099398 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099401 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099403 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099405 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099408 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099410 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099412 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099414 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099417 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099419 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099422 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099424 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099427 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099430 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099432 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099435 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:53.111015 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099437 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099439 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099441 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099444 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099446 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099448 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099451 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099454 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099457 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099461 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099463 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099465 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099468 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099470 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099473 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099475 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099477 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099480 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099482 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:53.111480 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099487 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099490 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099492 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099495 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099498 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099501 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099503 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099506 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099508 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099511 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099513 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099516 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099518 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099522 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099526 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099528 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099532 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099535 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099537 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:53.111938 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099540 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099542 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099545 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099547 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099550 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099552 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099554 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099556 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099559 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099561 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099563 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099566 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099569 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099573 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.099595 6400 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099625 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:53.112472 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099629 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099632 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099634 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099637 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099640 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099642 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099645 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099647 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099649 6400 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099652 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099655 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099657 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099659 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099662 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099664 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099667 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099669 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099672 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099674 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099677 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:53.112885 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099679 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099681 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099684 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099686 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099689 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099691 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099693 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099696 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099698 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099703 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099706 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099708 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099711 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099713 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099715 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099719 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099722 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099727 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099730 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:53.113368 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099732 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099735 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099737 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099740 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099742 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099745 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099747 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099750 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099752 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099754 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099757 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099759 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099762 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099764 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099767 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099769 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099772 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099774 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099777 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:53.113829 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099779 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099782 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099786 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099790 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099794 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099796 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099799 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099801 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099803 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099806 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099808 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099812 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099815 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099829 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099832 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099834 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099837 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099840 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099842 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:53.114282 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099844 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099847 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099850 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099853 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099855 6400 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099859 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099861 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099864 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099866 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099869 6400 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099871 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099874 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099877 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099879 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099882 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099884 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099888 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099890 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099893 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099896 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:53.114760 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099898 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099900 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099902 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099905 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099908 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099911 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099914 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:53.099916 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.099973 6400 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.101430 6400 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.101516 6400 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.101808 6400 server.go:1014] "Starting client certificate rotation" Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.102142 6400 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.102196 6400 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 23:59:39 +0000 UTC" deadline="2028-04-18 13:25:05.384080609 +0000 UTC" Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.102239 6400 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12781h19m12.281844513s" Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.102475 6400 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:05:53.115383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.103537 6400 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:05:53.116012 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.104843 6400 log.go:25] "Validated CRI v1 runtime API" Nov 03 00:05:53.116012 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.106756 6400 log.go:25] "Validated CRI v1 image API" Nov 03 00:05:53.116012 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.107501 6400 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 00:05:53.116012 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.110058 6400 fs.go:135] Filesystem UUIDs: map[794c63d6-6a4f-47cf-895c-07c128ca5477:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e99209dc-730c-4191-929e-252c875033b6:/dev/nvme0n1p3] Nov 03 00:05:53.116198 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.110080 6400 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/044a2e3307e37f67d0309483ddf7e14c808a18edae993e284963bbc7d33fcabf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/044a2e3307e37f67d0309483ddf7e14c808a18edae993e284963bbc7d33fcabf/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ad6c1d37cce5458301f9e4f1e8ebb0ceee5b80eb004af22325bf33b627cb883/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ad6c1d37cce5458301f9e4f1e8ebb0ceee5b80eb004af22325bf33b627cb883/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/348a3f58262f547f6b4d5e1c1a847ac135375ac640bec29138e6575a25c8ec8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/348a3f58262f547f6b4d5e1c1a847ac135375ac640bec29138e6575a25c8ec8e/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c70353946f43e9ce390ba9999746c9988a0b338c897bdfc31848c4e6c343335/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c70353946f43e9ce390ba9999746c9988a0b338c897bdfc31848c4e6c343335/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45140effbc3f1d941815a354f960bc0cf50243c7d77daabf71b2cff913846c35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45140effbc3f1d941815a354f960bc0cf50243c7d77daabf71b2cff913846c35/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/590267b131f92a1931738683b05c26f3488548e2948e1cfd6418d61d71bce1c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/590267b131f92a1931738683b05c26f3488548e2948e1cfd6418d61d71bce1c6/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67327af4b2141d01e717c838e10c38b51b72286ebf676a371297abe9bde098b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67327af4b2141d01e717c838e10c38b51b72286ebf676a371297abe9bde098b3/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73b46b35c4a79d74c453a6b455511d4c957f31a7f0b1aa3f20f7c37b7c2fc344/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73b46b35c4a79d74c453a6b455511d4c957f31a7f0b1aa3f20f7c37b7c2fc344/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79a3239bf922fe4e37346fd25a982ba6b37aa13b134ae1808e370f7390ed6e74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79a3239bf922fe4e37346fd25a982ba6b37aa13b134ae1808e370f7390ed6e74/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c1d91f20f7497418646f99c8223b2ad691358891c6cc6c73ae84e841d76e320/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c1d91f20f7497418646f99c8223b2ad691358891c6cc6c73ae84e841d76e320/userdata/shm major:0 minor:52 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/806034f7a24d9e9dd18667d2c50a1a57131ab581d5b4d0bb53b4b3629a468fc7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/806034f7a24d9e9dd18667d2c50a1a57131ab581d5b4d0bb53b4b3629a468fc7/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82d46560ca8e32594be1a17cf469209e0363dc04f76c7d543a3fd843f5731b0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82d46560ca8e32594be1a17cf469209e0363dc04f76c7d543a3fd843f5731b0c/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ced7c0f225c21078f70aa0d93ae5c09b38e6e4ba40773e95b3e812d45405c33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ced7c0f225c21078f70aa0d93ae5c09b38e6e4ba40773e95b3e812d45405c33/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a26db85a1fec77b5424925f7f25462f608ccc68b6ce178f8665b510ac870f91b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a26db85a1fec77b5424925f7f25462f608ccc68b6ce178f8665b510ac870f91b/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3ce8bbd2363f3c2df2c0e5010f7fd5fae48adc55f438b6c2053fd078d7997d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3ce8bbd2363f3c2df2c0e5010f7fd5fae48adc55f438b6c2053fd078d7997d6/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1280e8938be75acaed69f833c5b3113547df5dd3eca07ce0d154d997bd8a551/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1280e8938be75acaed69f833c5b3113547df5dd3eca07ce0d154d997bd8a551/userdata/shm major:0 minor:378 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b48cb7337658a344140015fd5fd7043c04af3900043258a64a07b9b86be94398/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b48cb7337658a344140015fd5fd7043c04af3900043258a64a07b9b86be94398/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4a59b5627bd5b0f70bb3ad7d6fa09bcbf2054feed3c2ff050080564e3c924d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4a59b5627bd5b0f70bb3ad7d6fa09bcbf2054feed3c2ff050080564e3c924d3/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c50d49b9f82cd3ed2eec19257ca8e031400c195d5cfb22bf22e8daf3206f5ed6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c50d49b9f82cd3ed2eec19257ca8e031400c195d5cfb22bf22e8daf3206f5ed6/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc06d0690d91342e97c77ca5c6a01f5c686d40166348547a5d77f4236f3497f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc06d0690d91342e97c77ca5c6a01f5c686d40166348547a5d77f4236f3497f4/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df9902e0aac9259e1b20e7a2b76d199df705d2a00827f5821e434d7adc400151/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df9902e0aac9259e1b20e7a2b76d199df705d2a00827f5821e434d7adc400151/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8c73a3b0a3305e7ee8b22db5e28b8f2d526dfe05628c52bf487cba213a2a1fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8c73a3b0a3305e7ee8b22db5e28b8f2d526dfe05628c52bf487cba213a2a1fd/userdata/shm major:0 minor:397 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea7935e552c6446ca8941dff5f42f86474263ad910c86eed14b76f06739b0bbd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea7935e552c6446ca8941dff5f42f86474263ad910c86eed14b76f06739b0bbd/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6740cdf7088b72d34df1b4c08503fb08155e66c03b920e7dff823589e1863d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6740cdf7088b72d34df1b4c08503fb08155e66c03b920e7dff823589e1863d1/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f84132e38423833f69d0b21d3411745548cdca2b397efb770c472bfb053d06cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f84132e38423833f69d0b21d3411745548cdca2b397efb770c472bfb053d06cf/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/169c66f9-91af-4e46-bf67-019e0cec4a43/volumes/kubernetes.io~projected/kube-api-access-kxhkl:{mountpoint:/var/lib/kubelet/pods/169c66f9-91af-4e46-bf67-019e0cec4a43/volumes/kubernetes.io~projected/kube-api-access-kxhkl major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/169c66f9-91af-4e46-bf67-019e0cec4a43/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/169c66f9-91af-4e46-bf67-019e0cec4a43/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/169c66f9-91af-4e46-bf67-019e0cec4a43/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/169c66f9-91af-4e46-bf67-019e0cec4a43/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd/volumes/kubernetes.io~projected/kube-api-access-tz5nz:{mountpoint:/var/lib/kubelet/pods/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd/volumes/kubernetes.io~projected/kube-api-access-tz5nz major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/423516a7-579e-492a-93dc-0c21e3faa673/volumes/kubernetes.io~projected/kube-api-access-9g9zg:{mountpoint:/var/lib/kubelet/pods/423516a7-579e-492a-93dc-0c21e3faa673/volumes/kubernetes.io~projected/kube-api-access-9g9zg major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/423516a7-579e-492a-93dc-0c21e3faa673/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/423516a7-579e-492a-93dc-0c21e3faa673/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/423516a7-579e-492a-93dc-0c21e3faa673/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/423516a7-579e-492a-93dc-0c21e3faa673/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43785dfc-6e27-45e2-8608-5fec9dfa4a86/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/43785dfc-6e27-45e2-8608-5fec9dfa4a86/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43785dfc-6e27-45e2-8608-5fec9dfa4a86/volumes/kubernetes.io~projected/kube-api-access-28lwp:{mountpoint:/var/lib/kubelet/pods/43785dfc-6e27-45e2-8608-5fec9dfa4a86/volumes/kubernetes.io~projected/kube-api-access-28lwp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43785dfc-6e27-45e2-8608-5fec9dfa4a86/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/43785dfc-6e27-45e2-8608-5fec9dfa4a86/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/443202b2-a46c-41af-b097-93658ee871f8/volumes/kubernetes.io~projected/kube-api-access-j6lrg:{mountpoint:/var/lib/kubelet/pods/443202b2-a46c-41af-b097-93658ee871f8/volumes/kubernetes.io~projected/kube-api-access-j6lrg major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/443202b2-a46c-41af-b097-93658ee871f8/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/443202b2-a46c-41af-b097-93658ee871f8/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4aa46084-b411-4ee1-98bb-145b0e8438c7/volumes/kubernetes.io~projected/kube-api-access-wfxj7:{mountpoint:/var/lib/kubelet/pods/4aa46084-b411-4ee1-98bb-145b0e8438c7/volumes/kubernetes.io~projected/kube-api-access-wfxj7 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc/volumes/kubernetes.io~projected/kube-api-access-nc96d:{mountpoint:/var/lib/kubelet/pods/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc/volumes/kubernetes.io~projected/kube-api-access-nc96d major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5367d682-754b-4925-ae57-40a1246d15e2/volumes/kubernetes.io~projected/kube-api-access-pj8fg:{mountpoint:/var/lib/kubelet/pods/5367d682-754b-4925-ae57-40a1246d15e2/volumes/kubernetes.io~projected/kube-api-access-pj8fg major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b0b5df-4723-4f99-98a0-b7a386f3d0c5/volumes/kubernetes.io~projected/kube-api-access-c9mww:{mountpoint:/var/lib/kubelet/pods/54b0b5df-4723-4f99-98a0-b7a386f3d0c5/volumes/kubernetes.io~projected/kube-api-access-c9mww major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b0b5df-4723-4f99-98a0-b7a386f3d0c5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/54b0b5df-4723-4f99-98a0-b7a386f3d0c5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b0b5df-4723-4f99-98a0-b7a386f3d0c5/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/54b0b5df-4723-4f99-98a0-b7a386f3d0c5/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58742d3c-ca4d-424e-aec7-35661094ad41/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/58742d3c-ca4d-424e-aec7-35661094ad41/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58742d3c-ca4d-424e-aec7-35661094ad41/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/58742d3c-ca4d-424e-aec7-35661094ad41/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58742d3c-ca4d-424e-aec7-35661094ad41/volumes/kubernetes.io~projected/kube-api-access-clcmb:{mountpoint:/var/lib/kubelet/pods/58742d3c-ca4d-424e-aec7-35661094ad41/volumes/kubernetes.io~projected/kube-api-access-clcmb major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0/volumes/kubernetes.io~projected/kube-api-access-gp5bx:{mountpoint:/var/lib/kubelet/pods/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0/volumes/kubernetes.io~projected/kube-api-access-gp5bx major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7314649e-0fd8-4bcd-bcfa-5140fb2711cf/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7314649e-0fd8-4bcd-bcfa-5140fb2711cf/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86519442-21ae-43dc-98bc-730f8212e34b/volumes/kubernetes.io~projected/kube-api-access-95txv:{mountpoint:/var/lib/kubelet/pods/86519442-21ae-43dc-98bc-730f8212e34b/volumes/kubernetes.io~projected/kube-api-access-95txv major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86519442-21ae-43dc-98bc-730f8212e34b/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/86519442-21ae-43dc-98bc-730f8212e34b/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ba97f37-1485-4215-abe3-3c9083759d8a/volumes/kubernetes.io~projected/kube-api-access-l8664:{mountpoint:/var/lib/kubelet/pods/8ba97f37-1485-4215-abe3-3c9083759d8a/volumes/kubernetes.io~projected/kube-api-access-l8664 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ba97f37-1485-4215-abe3-3c9083759d8a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8ba97f37-1485-4215-abe3-3c9083759d8a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ba97f37-1485-4215-abe3-3c9083759d8a/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/8ba97f37-1485-4215-abe3-3c9083759d8a/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9967e995-dc16-4869-bb2f-02eb340eba3e/volumes/kubernetes.io~projected/kube-api-access-xgv7z:{mountpoint:/var/lib/kubelet/pods/9967e995-dc16-4869-bb2f-02eb340eba3e/volumes/kubernetes.io~projected/kube-api-access-xgv7z major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9967e995-dc16-4869-bb2f-02eb340eba3e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9967e995-dc16-4869-bb2f-02eb340eba3e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9967e995-dc16-4869-bb2f-02eb340eba3e/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/9967e995-dc16-4869-bb2f-02eb340eba3e/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ee339b2-b471-4efd-829e-dd247c78685a/volumes/kubernetes.io~projected/kube-api-access-x7qzc:{mountpoint:/var/lib/kubelet/pods/9ee339b2-b471-4efd-829e-dd247c78685a/volumes/kubernetes.io~projected/kube-api-access-x7qzc major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ee339b2-b471-4efd-829e-dd247c78685a/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/9ee339b2-b471-4efd-829e-dd247c78685a/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ee339b2-b471-4efd-829e-dd247c78685a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/9ee339b2-b471-4efd-829e-dd247c78685a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ee339b2-b471-4efd-829e-dd247c78685a/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/9ee339b2-b471-4efd-829e-dd247c78685a/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b962e9e5-781e-450c-8c3f-17e217c67df6/volumes/kubernetes.io~projected/kube-api-access-sqgv8:{mountpoint:/var/lib/kubelet/pods/b962e9e5-781e-450c-8c3f-17e217c67df6/volumes/kubernetes.io~projected/kube-api-access-sqgv8 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b962e9e5-781e-450c-8c3f-17e217c67df6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/b962e9e5-781e-450c-8c3f-17e217c67df6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~projected/kube-api-access-pl4rl:{mountpoint:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~projected/kube-api-access-pl4rl major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5d41c02-03bc-4840-ae2f-a1590dda4ee9/volumes/kubernetes.io~projected/kube-api-access-df88n:{mountpoint:/var/lib/kubelet/pods/d5d41c02-03bc-4840-ae2f-a1590dda4ee9/volumes/kubernetes.io~projected/kube-api-access-df88n major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~projected/kube-api-access-ztn7x:{mountpoint:/var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~projected/kube-api-access-ztn7x major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddae2ef2-bac4-44c3-bef2-0d40c705c141/volumes/kubernetes.io~projected/kube-api-access-z9pj8:{mountpoint:/var/lib/kubelet/pods/ddae2ef2-bac4-44c3-bef2-0d40c705c141/volumes/kubernetes.io~projected/kube-api-access-z9pj8 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea31369b-b8c9-4405-ab0c-6796cd41cd7a/volumes/kubernetes.io~projected/kube-api-access-jg8sg:{mountpoint:/var/lib/kubelet/pods/ea31369b-b8c9-4405-ab0c-6796cd41cd7a/volumes/kubernetes.io~projected/kube-api-access-jg8sg major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea31369b-b8c9-4405-ab0c-6796cd41cd7a/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/ea31369b-b8c9-4405-ab0c-6796cd41cd7a/volumes/kubernetes.io~secret/metrics-certs major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd761dae-da21-44e4-adad-287a5d204bd3/volumes/kubernetes.io~projected/kube-api-access-cxrpd:{mountpoint:/var/lib/kubelet/pods/fd761dae-da21-44e4-adad-287a5d204bd3/volumes/kubernetes.io~projected/kube-api-access-cxrpd major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdf82aaa-e340-4a97-a3f4-297694eccab3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fdf82aaa-e340-4a97-a3f4-297694eccab3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:355 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/15a0da73e482237d20d94a215289cdb7b9c8ed9392ea064c9dfe99c1d7cc95a3/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8bf9a6fab27be392008c218cd788e4ee1285c551b149faa50eff6e84a1be4998/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/68c68ab99f5e507446777a2353f9e53b3dacdc25922b0b1e75f182bdf4dca199/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ece4d4912083e32c9c6e7634697d814bb078b5da550afdd98aa7b6e1064dd504/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/336f2b9cbded4ff778fdbcd8fa1070bbc6d588a71b35e4aaf53424fe5e8627ae/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d151e63f8d4084fdd5aa0f352408f280f43e1e3756b37b0334c6f5b66522d80d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d106e0eed7615e631edfd5ddbd8eba58a2ae75bc6026852fb75a3b32144e6a30/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/af078353abf9fbbf0e48472a715085e155e2cf2e66e5603efb6a8df0fc145c21/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/850cfbc08a614c08e9bdbfc175b6bda82534ef2b8aaa317c98dfa5e4577265a5/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/e9deaa728dd52d9f12b4b27e2279d5acf94ae2369ae216ed1b797ee17010421d/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/d26466039309105e16d12a2be60ffe7e50e9ea3406207205e890ca20d13a0e13/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/58166015b7bd763af58db8a09741dc715f46af342db431c2318c86dc65f53f15/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/92d6e37df34323914134edfb3e1aa26539ae318f3b5204a46551316782d792d9/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5b09d35e15cf900d8e095542841e2abd20dfdfb2cce08fcffd20edda51ecbbc8/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5d52e6a76c18080aac9f32b261d72286bc38e8fcee431e7d6846ce0adea7e61a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/278acb8e74c5c888b494052e4c6d885afa15a2bc06d6669acdfda3b6408d2c3d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/17daf2257dad3459a4c695a6ed05408bd316b22222b1ebff8676e3768a9d0a01/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/819fe6feed9b54d9972e597b936b35de1ae111bfb8261c0a1c6ab88901da3db6/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8476830d057c5db3aa8f1228cbcac2cdb65d9bdae5c8251f4ed4d5e7cec2bcb3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/8d15aa57ce2f5a39dc54dd40420ddf94e4a32835fedf7fc270d230d0176cdb4c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4c575d3831e023d5dee0e9e4e6a5e3193de584539bcb0d64e50784e25c47c45c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/543efe2c8634268b9fd0382d305f9704213f85992920c05d9448bbdcece32511/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/4f1dec05aec77a1a810b4c400bd12dd532ae95ba8c4626a1e12132210fef31e1/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/0b206b975f10d0e76fbf6171471f7a9f11d9cb36d4dee9d3b6cceaf185874479/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/3b90763d4cf4a120a7f4d0efb5eb5ab248fe3b4a67d9699ee09bba580adf5eb2/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/4acb569b8b85a094f7cee038fa8c98b7d9a2d46c1b9717881b23a4fcf77d5bc0/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/ee071e4329cb5ed24623380f6aa8886bcbe7ecdb8242e82deee0c29f9bca0468/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/c9ec674b489b34c4f3bc6d983bc87bc9f04b212fbb9823b8e24428a5b40b2dea/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/d0f796203f7d300cd3f2130dc474f15c56a9955a7de991eaa129cf2229f53bd4/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/dd95b8feb5901f37be0807d8b42c3a55a0c210c2b7ec663c353b150215869094/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/5ce8fc621a83b440d3a13b434c715491dbf8297edd2cb7dcf52ecfac84145cd5/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/fbc41aee9220336cbdf8f628f90285effab63098eb0ad6c39bc5b18e1717e7f7/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/f12ab20b1c12e70b80b769d353855d3efd73704fdf1ef7247dbfd6358003314b/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/30440079a87e87c8263a34b9d55503f932869fb845b9ea69c08132cbe0e72749/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/14c76447a9dd68615a484f4b473e9405eea60c22f22776539eb3a6c75920c079/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/1ea531a5d4ec815cd4b507a6d98741a59e2cdd3007d3ba2d8153f35422698be6/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/7856457ccd8dac52722bdcb6f017b7ef140f397da9132d3be814aaa7e73344e3/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/cc61018faa160bc7afa71a9a348375b287ee292578cd04ac9306aac8add09d1b/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/8ff55a145d033cd8cd3210d4b7e8acbb285cd448fe15dd5b9f7103ac66b3d8fb/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/7fc6e9e86eb0f6ce80b06d5652204c45c04a1873b8c5eec2d780c83dc410450b/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/3c67c1f405c213a81c053c3835ba73ac799b86cbcc4650a250c8bee725ef65b1/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/a4c6520b74086384867d71b56200b1d2a3329192b4e8af1b45ff0057185199a6/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/a51797ddf6c42b728b1f90616b1de86d01ccf6aad166a6b8f28810184d1cc4dd/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/9afeb2713f8c1aa9b224187a319da45bc56c0d2636fb194092d8eed20f43b8c3/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/d868ed7ef1fcd5f9df85de91bdef1c72619c0eb7d0035b74467f6211eb95bbeb/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/a55c80a445e8d4e415a0532fcb5d19c03a2a5fcac6f0e63798b49d95a8b0a6e6/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/792ce54b3bad6a3b5986795627cf72efc739dd41da7b459f9f26610796bbdce5/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/5b955759f4998d861a3dc7134371393c6ef0f68a5699a5216741931694d3c7d9/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/0a9b40b0651c80f8c8e47b19d9d0eeb8ea516ce822ad25b7ad2de475cae43e15/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/a36b56ba92d237fe32494345924318a415b1ef2cb77e606a2fd9483deab224d8/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/bc53e809ca563a34ea113ea211d1be559ac56575b4e4965b68958b583dabdcc7/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/5a35fee2b67117da80a7119c8a3499a75cbea214171dbfdb0390d9f1fa944efb/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/149f1d4471f6372fd7fb52c35b91aaf64bd684147205b28aca7bd9c3152510ef/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/e1136717d55b929f7a6085024a740663c6da29a546321d1ace684590a1b70622/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/4a98ec05c714f505a08b3496e0f70213d7952cdde6c3c71a7ae4e5eef6b6bbfe/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/7f8eb79b09c754ba53b22ace28d6c80c2aa6423bac8512fb06d63689f3b36608/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/06d7e72a5d0caf96d940b16031af72ca9766bbe2ddeee4b376d629e0e2ef3b41/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/6f70dc15d6346c5b9b7bd0af20e0463b10ee6cf7c2ff92ca8592371cad27d6f6/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/9f367a42da124f01f4f04ea891b65b5adbeed4544c2456e9d7fdc0a27316657a/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/3cdcd7d52ecab3b37af52f25f1b60c960184e988fcbc777f14dce37b9d746160/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/104143a06b4b3816eb456ab5a07291bc16337288e4c0f60ade3178f3f8bbad07/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/bdf991d1ba576ea54e1f35799c0e73543eb191051f47a4a0b6fa6d25bb3cfded/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-439:{mountpoint:/var/lib/containers/storage/overlay/7548bce638b0fbce1ae029b411aecaa846e8219f1b9a9bed5afdea872e6a0fa5/merged major:0 minor:439 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/aab5f5f856793d5cf3be4cae31f4ffcf2049da5592ea62a499105cef07eccb4d/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/115cc25e7f932442a7194367f8c67190ed99800f34cf6dcf075b9221ad56d588/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/e096c14d1696feccef8f04bc73f9029b847d5a1e00f2e32ef4c68ed7847115a1/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-461:{mountpoint:/var/lib/containers/storage/overlay/27b1be82a33a3278dfbf8c6e7a0ad2451471c68dee32dcd465201f4ada9b51a3/merged major:0 minor:461 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/5c6786f222e854519d9098ddd4a80ba8d155e326713178de1ed66cd1724537b8/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/d394ac26ddb266fd5fd3e9d3fdefd4c136701a7d4b9e405477aa4f2e5f928dd4/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/2e35958afff22f12a1d052c6d1f09a897b33162f4f511280a56e8791df64bb83/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b057fc088ca7c66ad906c4d438115fbb49133390c3730c4c78e49bd71b6c8f82/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/396cebbd8cbe285b9c466fd66ed46ebc4b5817cff3c4e17b047db047749af3fd/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/6a09d3ed0480ef3686a5a8ed026354e55c37407130389eec0df938c4f0b4202b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/2467fe81f94ddb0b1b9de42f0008dd1455cabb5051b97ac25bcf57a83eeb265d/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/ea8154d5f718d2aa0c5054ea73c71aa62a5a83c7e6cefe380110720db512fe86/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/084bd71002596ba70b4cccb1b81815e48016ba628962e3d65de347c0da09b73f/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/344aaa50dec4aaa7d96e361a932486590e3b3fff482898ae92f88927462fcf06/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/77979a03e6906efeba374f570dfb36a4084485cb4ba9cbbe5855068278b0b605/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 03 00:05:53.119321 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.118908 6400 manager.go:217] Machine: {Timestamp:2026-11-03 00:05:53.118438071 +0000 UTC m=+0.069378218 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199450 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25ef4360106dd448bd3188962a8553 SystemUUID:ec25ef43-6010-6dd4-48bd-3188962a8553 BootID:46ad9331-ebf9-478e-9986-d46b099757f4 Filesystems:[{Device:/var/lib/kubelet/pods/8ba97f37-1485-4215-abe3-3c9083759d8a/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79a3239bf922fe4e37346fd25a982ba6b37aa13b134ae1808e370f7390ed6e74/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8ba97f37-1485-4215-abe3-3c9083759d8a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:285 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea31369b-b8c9-4405-ab0c-6796cd41cd7a/volumes/kubernetes.io~projected/kube-api-access-jg8sg DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df9902e0aac9259e1b20e7a2b76d199df705d2a00827f5821e434d7adc400151/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:386 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ddae2ef2-bac4-44c3-bef2-0d40c705c141/volumes/kubernetes.io~projected/kube-api-access-z9pj8 DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc/volumes/kubernetes.io~projected/kube-api-access-nc96d DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c1d91f20f7497418646f99c8223b2ad691358891c6cc6c73ae84e841d76e320/userdata/shm DeviceMajor:0 DeviceMinor:52 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/423516a7-579e-492a-93dc-0c21e3faa673/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:257 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:331 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/169c66f9-91af-4e46-bf67-019e0cec4a43/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:244 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/54b0b5df-4723-4f99-98a0-b7a386f3d0c5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8ba97f37-1485-4215-abe3-3c9083759d8a/volumes/kubernetes.io~projected/kube-api-access-l8664 DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ee339b2-b471-4efd-829e-dd247c78685a/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:372 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4aa46084-b411-4ee1-98bb-145b0e8438c7/volumes/kubernetes.io~projected/kube-api-access-wfxj7 DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ced7c0f225c21078f70aa0d93ae5c09b38e6e4ba40773e95b3e812d45405c33/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/169c66f9-91af-4e46-bf67-019e0cec4a43/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/348a3f58262f547f6b4d5e1c1a847ac135375ac640bec29138e6575a25c8ec8e/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c70353946f43e9ce390ba9999746c9988a0b338c897bdfc31848c4e6c343335/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea31369b-b8c9-4405-ab0c-6796cd41cd7a/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:356 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-439 DeviceMajor:0 DeviceMinor:439 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/423516a7-579e-492a-93dc-0c21e3faa673/volumes/kubernetes.io~projected/kube-api-access-9g9zg DeviceMajor:0 DeviceMinor:259 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9967e995-dc16-4869-bb2f-02eb340eba3e/volumes/kubernetes.io~projected/kube-api-access-xgv7z DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43785dfc-6e27-45e2-8608-5fec9dfa4a86/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58742d3c-ca4d-424e-aec7-35661094ad41/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67327af4b2141d01e717c838e10c38b51b72286ebf676a371297abe9bde098b3/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a26db85a1fec77b5424925f7f25462f608ccc68b6ce178f8665b510ac870f91b/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5367d682-754b-4925-ae57-40a1246d15e2/volumes/kubernetes.io~projected/kube-api-access-pj8fg DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/58742d3c-ca4d-424e-aec7-35661094ad41/volumes/kubernetes.io~projected/kube-api-access-clcmb DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82d46560ca8e32594be1a17cf469209e0363dc04f76c7d543a3fd843f5731b0c/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1280e8938be75acaed69f833c5b3113547df5dd3eca07ce0d154d997bd8a551/userdata/shm DeviceMajor:0 DeviceMinor:378 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:384 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-461 DeviceMajor:0 DeviceMinor:461 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:336 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b48cb7337658a344140015fd5fd7043c04af3900043258a64a07b9b86be94398/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4a59b5627bd5b0f70bb3ad7d6fa09bcbf2054feed3c2ff050080564e3c924d3/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/43785dfc-6e27-45e2-8608-5fec9dfa4a86/volumes/kubernetes.io~projected/kube-api-access-28lwp DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd761dae-da21-44e4-adad-287a5d204bd3/volumes/kubernetes.io~projected/kube-api-access-cxrpd DeviceMajor:0 DeviceMinor:361 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b962e9e5-781e-450c-8c3f-17e217c67df6/volumes/kubernetes.io~projected/kube-api-access-sqgv8 DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0/volumes/kubernetes.io~projected/kube-api-access-gp5bx DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ad6c1d37cce5458301f9e4f1e8ebb0ceee5b80eb004af22325bf33b627cb883/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/54b0b5df-4723-4f99-98a0-b7a386f3d0c5/volumes/kubernetes.io~projected/kube-api-access-c9mww DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fdf82aaa-e340-4a97-a3f4-297694eccab3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:355 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45140effbc3f1d941815a354f960bc0cf50243c7d77daabf71b2cff913846c35/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6740cdf7088b72d34df1b4c08503fb08155e66c03b920e7dff823589e1863d1/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86519442-21ae-43dc-98bc-730f8212e34b/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/423516a7-579e-492a-93dc-0c21e3faa673/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:258 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3ce8bbd2363f3c2df2c0e5010f7fd5fae48adc55f438b6c2053fd078d7997d6/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b962e9e5-781e-450c-8c3f-17e217c67df6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/590267b131f92a1931738683b05c26f3488548e2948e1cfd6418d61d71bce1c6/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9967e995-dc16-4869-bb2f-02eb340eba3e/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~projected/kube-api-access-pl4rl DeviceMajor:0 DeviceMinor:342 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:385 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/7314649e-0fd8-4bcd-bcfa-5140fb2711cf/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c50d49b9f82cd3ed2eec19257ca8e031400c195d5cfb22bf22e8daf3206f5ed6/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/54b0b5df-4723-4f99-98a0-b7a386f3d0c5/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/044a2e3307e37f67d0309483ddf7e14c808a18edae993e284963bbc7d33fcabf/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/169c66f9-91af-4e46-bf67-019e0cec4a43/volumes/kubernetes.io~projected/kube-api-access-kxhkl DeviceMajor:0 DeviceMinor:252 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ee339b2-b471-4efd-829e-dd247c78685a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:374 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43785dfc-6e27-45e2-8608-5fec9dfa4a86/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86519442-21ae-43dc-98bc-730f8212e34b/volumes/kubernetes.io~projected/kube-api-access-95txv DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:387 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/58742d3c-ca4d-424e-aec7-35661094ad41/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea7935e552c6446ca8941dff5f42f86474263ad910c86eed14b76f06739b0bbd/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73b46b35c4a79d74c453a6b455511d4c957f31a7f0b1aa3f20f7c37b7c2fc344/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/443202b2-a46c-41af-b097-93658ee871f8/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:370 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9967e995-dc16-4869-bb2f-02eb340eba3e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f84132e38423833f69d0b21d3411745548cdca2b397efb770c472bfb053d06cf/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/806034f7a24d9e9dd18667d2c50a1a57131ab581d5b4d0bb53b4b3629a468fc7/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cbec544b-d079-48c5-9d21-0e739d7512d4/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:337 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9ee339b2-b471-4efd-829e-dd247c78685a/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:373 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd/volumes/kubernetes.io~projected/kube-api-access-tz5nz DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc06d0690d91342e97c77ca5c6a01f5c686d40166348547a5d77f4236f3497f4/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ee339b2-b471-4efd-829e-dd247c78685a/volumes/kubernetes.io~projected/kube-api-access-x7qzc DeviceMajor:0 DeviceMinor:377 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d648f173-aa93-4fe7-999b-b50c88be3440/volumes/kubernetes.io~projected/kube-api-access-ztn7x DeviceMajor:0 DeviceMinor:388 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8c73a3b0a3305e7ee8b22db5e28b8f2d526dfe05628c52bf487cba213a2a1fd/userdata/shm DeviceMajor:0 DeviceMinor:397 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5d41c02-03bc-4840-ae2f-a1590dda4ee9/volumes/kubernetes.io~projected/kube-api-access-df88n DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/443202b2-a46c-41af-b097-93658ee871f8/volumes/kubernetes.io~projected/kube-api-access-j6lrg DeviceMajor:0 DeviceMinor:371 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:044a2e3307e37f6 MacAddress:f6:b1:3e:f4:b2:7a Speed:10000 Mtu:8901} {Name:0ad6c1d37cce545 MacAddress:12:e5:dd:d6:41:47 Speed:10000 Mtu:8901} {Name:348a3f58262f547 MacAddress:ae:f3:9f:99:f5:76 Speed:10000 Mtu:8901} {Name:590267b131f92a1 MacAddress:fa:22:27:44:13:35 Speed:10000 Mtu:8901} {Name:73b46b35c4a79d7 MacAddress:5a:cc:f2:33:28:c8 Speed:10000 Mtu:8901} {Name:79a3239bf922fe4 MacAddress:6a:65:31:29:a7:bc Speed:10000 Mtu:8901} {Name:806034f7a24d9e9 MacAddress:ba:f9:fe:9b:bd:3f Speed:10000 Mtu:8901} {Name:82d46560ca8e325 MacAddress:82:c3:2e:2e:44:50 Speed:10000 Mtu:8901} {Name:a26db85a1fec77b MacAddress:3e:9f:4d:37:78:93 Speed:10000 Mtu:8901} {Name:b1280e8938be75a MacAddress:26:aa:b6:cd:fa:bb Speed:10000 Mtu:8901} {Name:b48cb7337658a34 MacAddress:5e:9b:98:73:1b:38 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:03:56:a8:b1:f9 Speed:0 Mtu:9001} {Name:br-int MacAddress:66:93:46:1d:93:aa Speed:0 Mtu:8901} {Name:e8c73a3b0a3305e MacAddress:5e:31:6f:a7:87:c8 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:03:56:a8:b1:f9 Speed:0 Mtu:9001} {Name:f6740cdf7088b72 MacAddress:96:c5:d0:5a:ea:95 Speed:10000 Mtu:8901} {Name:f84132e38423833 MacAddress:6a:8c:04:f8:d0:d6 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:66:2e:31:76:7d:0f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3e:d2:06:bb:ff:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 00:05:53.119560 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.119325 6400 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 00:05:53.119560 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.119369 6400 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 00:05:53.119690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.119666 6400 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 00:05:53.119811 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.119692 6400 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-199.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 00:05:53.119866 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.119814 6400 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 00:05:53.119866 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.119835 6400 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 00:05:53.119866 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.119846 6400 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:05:53.119866 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.119859 6400 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:05:53.119974 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.119947 6400 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 00:05:53.120129 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.120119 6400 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 00:05:53.120242 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.120233 6400 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 00:05:53.120280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.120271 6400 kubelet.go:493] "Attempting to sync node with API server" Nov 03 00:05:53.120307 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.120291 6400 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 00:05:53.120307 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.120302 6400 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 00:05:53.120363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.120311 6400 kubelet.go:405] "Adding apiserver pod source" Nov 03 00:05:53.120363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.120319 6400 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 00:05:53.121250 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.121235 6400 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 00:05:53.121286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.121254 6400 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:05:53.121747 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.121727 6400 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 00:05:53.124440 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124425 6400 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 00:05:53.124440 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124439 6400 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:05:53.124541 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124525 6400 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 00:05:53.124699 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124689 6400 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 00:05:53.124728 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124717 6400 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 00:05:53.124813 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124804 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 00:05:53.124859 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124839 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 00:05:53.124859 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124845 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 00:05:53.124859 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124851 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 00:05:53.124859 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124857 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 00:05:53.124965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124864 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 00:05:53.124965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124870 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 00:05:53.124965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124875 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 00:05:53.124965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124882 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 00:05:53.124965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124887 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 00:05:53.124965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124901 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 00:05:53.124965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124911 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 00:05:53.124965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124936 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 00:05:53.124965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.124942 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 00:05:53.128321 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.128277 6400 server.go:1263] "Started kubelet" Nov 03 00:05:53.128488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.128463 6400 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 00:05:53.128690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.128599 6400 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 00:05:53.128690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.128650 6400 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 00:05:53.129059 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.129035 6400 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 00:05:53.129296 ip-10-0-135-199 systemd[1]: Started Kubernetes Kubelet. Nov 03 00:05:53.129767 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.129751 6400 server.go:317] "Adding debug handlers to kubelet server" Nov 03 00:05:53.135736 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.135551 6400 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 00:05:53.135736 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.135575 6400 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 00:05:53.135736 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.135630 6400 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:40 +0000 UTC" deadline="2028-04-28 16:45:38.972022821 +0000 UTC" Nov 03 00:05:53.135736 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.135650 6400 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13024h39m45.836376676s" Nov 03 00:05:53.135736 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.135681 6400 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 00:05:53.135736 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.135694 6400 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 00:05:53.135736 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:05:53.135693 6400 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-199.us-west-1.compute.internal\" not found" Nov 03 00:05:53.136510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.135779 6400 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.137496 6400 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.137515 6400 factory.go:55] Registering systemd factory Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.137524 6400 factory.go:223] Registration of the systemd container factory successfully Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.137716 6400 factory.go:153] Registering CRI-O factory Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.137726 6400 factory.go:223] Registration of the crio container factory successfully Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.137744 6400 factory.go:103] Registering Raw factory Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.137758 6400 manager.go:1199] Started watching for new ooms in manager Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138119 6400 manager.go:319] Starting recovery of all containers Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138247 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d648f173-aa93-4fe7-999b-b50c88be3440" volumeName="kubernetes.io/projected/d648f173-aa93-4fe7-999b-b50c88be3440-kube-api-access-ztn7x" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138281 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" volumeName="kubernetes.io/projected/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-kube-api-access-jg8sg" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138295 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea31369b-b8c9-4405-ab0c-6796cd41cd7a" volumeName="kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138308 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35dc422a-ae29-4b68-b3fc-efd9b44fa2dd" volumeName="kubernetes.io/projected/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-kube-api-access-tz5nz" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138320 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b0b5df-4723-4f99-98a0-b7a386f3d0c5" volumeName="kubernetes.io/secret/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138333 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b962e9e5-781e-450c-8c3f-17e217c67df6" volumeName="kubernetes.io/secret/b962e9e5-781e-450c-8c3f-17e217c67df6-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138345 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbec544b-d079-48c5-9d21-0e739d7512d4" volumeName="kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138355 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" volumeName="kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138368 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="423516a7-579e-492a-93dc-0c21e3faa673" volumeName="kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-serving-cert" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138380 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ba97f37-1485-4215-abe3-3c9083759d8a" volumeName="kubernetes.io/configmap/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138392 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d648f173-aa93-4fe7-999b-b50c88be3440" volumeName="kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-federate-client-tls" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138404 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="423516a7-579e-492a-93dc-0c21e3faa673" volumeName="kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-oauth-serving-cert" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138415 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ba97f37-1485-4215-abe3-3c9083759d8a" volumeName="kubernetes.io/configmap/8ba97f37-1485-4215-abe3-3c9083759d8a-metrics-client-ca" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138426 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" volumeName="kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138440 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddae2ef2-bac4-44c3-bef2-0d40c705c141" volumeName="kubernetes.io/projected/ddae2ef2-bac4-44c3-bef2-0d40c705c141-kube-api-access-z9pj8" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138453 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="423516a7-579e-492a-93dc-0c21e3faa673" volumeName="kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-service-ca" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138465 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="423516a7-579e-492a-93dc-0c21e3faa673" volumeName="kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-oauth-config" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138477 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86519442-21ae-43dc-98bc-730f8212e34b" volumeName="kubernetes.io/configmap/86519442-21ae-43dc-98bc-730f8212e34b-config-volume" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138491 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9967e995-dc16-4869-bb2f-02eb340eba3e" volumeName="kubernetes.io/configmap/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138503 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9967e995-dc16-4869-bb2f-02eb340eba3e" volumeName="kubernetes.io/projected/9967e995-dc16-4869-bb2f-02eb340eba3e-kube-api-access-xgv7z" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138514 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ee339b2-b471-4efd-829e-dd247c78685a" volumeName="kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-secret-metrics-server-tls" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138526 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="169c66f9-91af-4e46-bf67-019e0cec4a43" volumeName="kubernetes.io/configmap/169c66f9-91af-4e46-bf67-019e0cec4a43-metrics-client-ca" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138540 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="423516a7-579e-492a-93dc-0c21e3faa673" volumeName="kubernetes.io/projected/423516a7-579e-492a-93dc-0c21e3faa673-kube-api-access-9g9zg" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138553 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9967e995-dc16-4869-bb2f-02eb340eba3e" volumeName="kubernetes.io/secret/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138563 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ee339b2-b471-4efd-829e-dd247c78685a" volumeName="kubernetes.io/configmap/9ee339b2-b471-4efd-829e-dd247c78685a-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138574 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d648f173-aa93-4fe7-999b-b50c88be3440" volumeName="kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-telemeter-client-tls" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138585 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd761dae-da21-44e4-adad-287a5d204bd3" volumeName="kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138597 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43785dfc-6e27-45e2-8608-5fec9dfa4a86" volumeName="kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-env-overrides" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138635 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b0b5df-4723-4f99-98a0-b7a386f3d0c5" volumeName="kubernetes.io/projected/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-kube-api-access-c9mww" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138651 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9967e995-dc16-4869-bb2f-02eb340eba3e" volumeName="kubernetes.io/empty-dir/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-textfile" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138664 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbec544b-d079-48c5-9d21-0e739d7512d4" volumeName="kubernetes.io/configmap/cbec544b-d079-48c5-9d21-0e739d7512d4-metrics-client-ca" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138675 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddae2ef2-bac4-44c3-bef2-0d40c705c141" volumeName="kubernetes.io/empty-dir/ddae2ef2-bac4-44c3-bef2-0d40c705c141-tmp-dir" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138689 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58742d3c-ca4d-424e-aec7-35661094ad41" volumeName="kubernetes.io/empty-dir/58742d3c-ca4d-424e-aec7-35661094ad41-tmp" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138701 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7314649e-0fd8-4bcd-bcfa-5140fb2711cf" volumeName="kubernetes.io/configmap/7314649e-0fd8-4bcd-bcfa-5140fb2711cf-konnectivity-ca" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138712 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d648f173-aa93-4fe7-999b-b50c88be3440" volumeName="kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138728 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="443202b2-a46c-41af-b097-93658ee871f8" volumeName="kubernetes.io/secret/443202b2-a46c-41af-b097-93658ee871f8-monitoring-plugin-cert" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138739 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b0b5df-4723-4f99-98a0-b7a386f3d0c5" volumeName="kubernetes.io/configmap/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-metrics-client-ca" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138751 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ba97f37-1485-4215-abe3-3c9083759d8a" volumeName="kubernetes.io/secret/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138764 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b962e9e5-781e-450c-8c3f-17e217c67df6" volumeName="kubernetes.io/empty-dir/b962e9e5-781e-450c-8c3f-17e217c67df6-data-volume" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138776 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43785dfc-6e27-45e2-8608-5fec9dfa4a86" volumeName="kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovnkube-config" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138787 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="443202b2-a46c-41af-b097-93658ee871f8" volumeName="kubernetes.io/projected/443202b2-a46c-41af-b097-93658ee871f8-kube-api-access-j6lrg" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138815 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5367d682-754b-4925-ae57-40a1246d15e2" volumeName="kubernetes.io/projected/5367d682-754b-4925-ae57-40a1246d15e2-kube-api-access-pj8fg" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138845 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0" volumeName="kubernetes.io/secret/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0-cert" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138856 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ee339b2-b471-4efd-829e-dd247c78685a" volumeName="kubernetes.io/configmap/9ee339b2-b471-4efd-829e-dd247c78685a-metrics-server-audit-profiles" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138867 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b962e9e5-781e-450c-8c3f-17e217c67df6" volumeName="kubernetes.io/configmap/b962e9e5-781e-450c-8c3f-17e217c67df6-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138882 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbec544b-d079-48c5-9d21-0e739d7512d4" volumeName="kubernetes.io/empty-dir/cbec544b-d079-48c5-9d21-0e739d7512d4-tmp" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138894 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="423516a7-579e-492a-93dc-0c21e3faa673" volumeName="kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-trusted-ca-bundle" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138905 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="423516a7-579e-492a-93dc-0c21e3faa673" volumeName="kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-console-config" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138917 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="423516a7-579e-492a-93dc-0c21e3faa673" volumeName="kubernetes.io/empty-dir/423516a7-579e-492a-93dc-0c21e3faa673-tmp" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138929 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ba97f37-1485-4215-abe3-3c9083759d8a" volumeName="kubernetes.io/projected/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-api-access-l8664" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138940 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9967e995-dc16-4869-bb2f-02eb340eba3e" volumeName="kubernetes.io/secret/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-tls" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138953 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbec544b-d079-48c5-9d21-0e739d7512d4" volumeName="kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-tls" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138963 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf82aaa-e340-4a97-a3f4-297694eccab3" volumeName="kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138977 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="169c66f9-91af-4e46-bf67-019e0cec4a43" volumeName="kubernetes.io/secret/169c66f9-91af-4e46-bf67-019e0cec4a43-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.138992 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b0b5df-4723-4f99-98a0-b7a386f3d0c5" volumeName="kubernetes.io/secret/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139003 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0" volumeName="kubernetes.io/projected/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0-kube-api-access-gp5bx" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139025 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7314649e-0fd8-4bcd-bcfa-5140fb2711cf" volumeName="kubernetes.io/secret/7314649e-0fd8-4bcd-bcfa-5140fb2711cf-agent-certs" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139038 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ba97f37-1485-4215-abe3-3c9083759d8a" volumeName="kubernetes.io/empty-dir/8ba97f37-1485-4215-abe3-3c9083759d8a-volume-directive-shadow" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139050 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ee339b2-b471-4efd-829e-dd247c78685a" volumeName="kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139064 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbec544b-d079-48c5-9d21-0e739d7512d4" volumeName="kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139076 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43785dfc-6e27-45e2-8608-5fec9dfa4a86" volumeName="kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovnkube-script-lib" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139088 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43785dfc-6e27-45e2-8608-5fec9dfa4a86" volumeName="kubernetes.io/secret/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139100 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4aa46084-b411-4ee1-98bb-145b0e8438c7" volumeName="kubernetes.io/configmap/4aa46084-b411-4ee1-98bb-145b0e8438c7-cni-binary-copy" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139112 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ee339b2-b471-4efd-829e-dd247c78685a" volumeName="kubernetes.io/empty-dir/9ee339b2-b471-4efd-829e-dd247c78685a-audit-log" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139125 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b962e9e5-781e-450c-8c3f-17e217c67df6" volumeName="kubernetes.io/projected/b962e9e5-781e-450c-8c3f-17e217c67df6-kube-api-access-sqgv8" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139138 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d648f173-aa93-4fe7-999b-b50c88be3440" volumeName="kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139150 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43785dfc-6e27-45e2-8608-5fec9dfa4a86" volumeName="kubernetes.io/projected/43785dfc-6e27-45e2-8608-5fec9dfa4a86-kube-api-access-28lwp" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139163 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4aa46084-b411-4ee1-98bb-145b0e8438c7" volumeName="kubernetes.io/configmap/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-daemon-config" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139174 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc" volumeName="kubernetes.io/projected/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-kube-api-access-nc96d" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139185 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86519442-21ae-43dc-98bc-730f8212e34b" volumeName="kubernetes.io/empty-dir/86519442-21ae-43dc-98bc-730f8212e34b-tmp-dir" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139200 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86519442-21ae-43dc-98bc-730f8212e34b" volumeName="kubernetes.io/projected/86519442-21ae-43dc-98bc-730f8212e34b-kube-api-access-95txv" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139213 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86519442-21ae-43dc-98bc-730f8212e34b" volumeName="kubernetes.io/secret/86519442-21ae-43dc-98bc-730f8212e34b-metrics-tls" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139226 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbec544b-d079-48c5-9d21-0e739d7512d4" volumeName="kubernetes.io/projected/cbec544b-d079-48c5-9d21-0e739d7512d4-kube-api-access-pl4rl" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139238 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbec544b-d079-48c5-9d21-0e739d7512d4" volumeName="kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139250 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4aa46084-b411-4ee1-98bb-145b0e8438c7" volumeName="kubernetes.io/projected/4aa46084-b411-4ee1-98bb-145b0e8438c7-kube-api-access-wfxj7" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139263 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5367d682-754b-4925-ae57-40a1246d15e2" volumeName="kubernetes.io/configmap/5367d682-754b-4925-ae57-40a1246d15e2-iptables-alerter-script" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139274 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ee339b2-b471-4efd-829e-dd247c78685a" volumeName="kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-client-ca-bundle" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139286 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" volumeName="kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cni-binary-copy" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139303 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d648f173-aa93-4fe7-999b-b50c88be3440" volumeName="kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139314 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d648f173-aa93-4fe7-999b-b50c88be3440" volumeName="kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-secret-telemeter-client" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139325 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="169c66f9-91af-4e46-bf67-019e0cec4a43" volumeName="kubernetes.io/secret/169c66f9-91af-4e46-bf67-019e0cec4a43-prometheus-operator-tls" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139338 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc" volumeName="kubernetes.io/configmap/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-serviceca" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139349 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58742d3c-ca4d-424e-aec7-35661094ad41" volumeName="kubernetes.io/empty-dir/58742d3c-ca4d-424e-aec7-35661094ad41-etc-tuned" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139362 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58742d3c-ca4d-424e-aec7-35661094ad41" volumeName="kubernetes.io/projected/58742d3c-ca4d-424e-aec7-35661094ad41-kube-api-access-clcmb" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139375 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ba97f37-1485-4215-abe3-3c9083759d8a" volumeName="kubernetes.io/secret/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139387 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbec544b-d079-48c5-9d21-0e739d7512d4" volumeName="kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139401 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" volumeName="kubernetes.io/projected/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-kube-api-access-df88n" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139412 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d648f173-aa93-4fe7-999b-b50c88be3440" volumeName="kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-metrics-client-ca" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139423 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="169c66f9-91af-4e46-bf67-019e0cec4a43" volumeName="kubernetes.io/projected/169c66f9-91af-4e46-bf67-019e0cec4a43-kube-api-access-kxhkl" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139436 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9967e995-dc16-4869-bb2f-02eb340eba3e" volumeName="kubernetes.io/configmap/9967e995-dc16-4869-bb2f-02eb340eba3e-metrics-client-ca" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139447 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ee339b2-b471-4efd-829e-dd247c78685a" volumeName="kubernetes.io/projected/9ee339b2-b471-4efd-829e-dd247c78685a-kube-api-access-x7qzc" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139459 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbec544b-d079-48c5-9d21-0e739d7512d4" volumeName="kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-grpc-tls" seLinuxMountContext="" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139471 6400 reconstruct.go:97] "Volume reconstruction finished" Nov 03 00:05:53.141411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.139478 6400 reconciler.go:29] "Reconciler: start to sync state" Nov 03 00:05:53.169864 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.169788 6400 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 00:05:53.171379 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.171359 6400 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 00:05:53.171379 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.171380 6400 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 00:05:53.171532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.171399 6400 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 00:05:53.171532 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:05:53.171449 6400 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 00:05:53.192368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.192328 6400 generic.go:356] "Generic (PLEG): container finished" podID="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" containerID="8ad37539a5307bb9b865264eb9949f64cf70e137c25999c1d707e92b19fc5870" exitCode=0 Nov 03 00:05:53.192368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.192365 6400 generic.go:356] "Generic (PLEG): container finished" podID="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" containerID="78cd4792fe2c55b193e89938eebbee01eda398559e183d0db2a74de260020a97" exitCode=0 Nov 03 00:05:53.192368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.192373 6400 generic.go:356] "Generic (PLEG): container finished" podID="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" containerID="2940ea85428cc337bf03b9b330e4e7b995b1b7d0b8edf296b2f5314cf568ab94" exitCode=0 Nov 03 00:05:53.192535 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.192379 6400 generic.go:356] "Generic (PLEG): container finished" podID="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" containerID="3d389ef9792e057fd1631434e746945c57aecb79e92c1b0dcc5e7e74718688d9" exitCode=0 Nov 03 00:05:53.192535 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.192385 6400 generic.go:356] "Generic (PLEG): container finished" podID="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" containerID="e92ca27e33798c244770b8fea0938d82fe39e56df0c27929301ad0d0f283522d" exitCode=0 Nov 03 00:05:53.192535 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.192389 6400 generic.go:356] "Generic (PLEG): container finished" podID="d5d41c02-03bc-4840-ae2f-a1590dda4ee9" containerID="ee8bce2c98bd0bb07d12e1315603b8ecc7cf683e53cf32ebce911934fb47051d" exitCode=0 Nov 03 00:05:53.206045 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.206010 6400 generic.go:356] "Generic (PLEG): container finished" podID="bdf730cee51b9b033223ceb412560cad" containerID="56df28890ee2e20e4617ba3c42b659c9055b084946299d3a316b187fd8160021" exitCode=0 Nov 03 00:05:53.213032 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.213008 6400 generic.go:356] "Generic (PLEG): container finished" podID="9967e995-dc16-4869-bb2f-02eb340eba3e" containerID="0330831cc4834e164c12da885a184fabfe851ddc493b0edc7c9f458dd1948b1e" exitCode=0 Nov 03 00:05:53.235975 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:05:53.235912 6400 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-199.us-west-1.compute.internal\" not found" Nov 03 00:05:53.256735 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.256715 6400 manager.go:324] Recovery completed Nov 03 00:05:53.272429 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:05:53.272397 6400 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 00:05:53.278230 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.278208 6400 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:05:53.281853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.280832 6400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:05:53.281853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.280868 6400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:05:53.281853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.280881 6400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:05:53.284432 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.284412 6400 cpu_manager.go:226] "Starting" policy="none" Nov 03 00:05:53.284432 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.284428 6400 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 00:05:53.284432 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.284430 6400 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:05:53.284583 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.284448 6400 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 00:05:53.284637 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.284602 6400 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 00:05:53.284637 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.284614 6400 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 00:05:53.284637 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.284631 6400 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 00:05:53.284637 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.284636 6400 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 00:05:53.284882 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.284642 6400 policy_none.go:50] "Start" Nov 03 00:05:53.284882 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.284653 6400 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 00:05:53.284882 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.284662 6400 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 00:05:53.285392 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.285378 6400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:05:53.285449 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.285418 6400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:05:53.285449 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.285429 6400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:05:53.285933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.285920 6400 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 00:05:53.285966 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.285936 6400 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 00:05:53.285966 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.285943 6400 policy_none.go:44] "Start" Nov 03 00:05:53.290505 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.290480 6400 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 00:05:53.292831 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.292805 6400 manager.go:341] "Starting Device Plugin manager" Nov 03 00:05:53.292899 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:05:53.292858 6400 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 00:05:53.292899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.292871 6400 server.go:87] "Starting device plugin registration server" Nov 03 00:05:53.293202 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.293191 6400 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 00:05:53.293235 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.293205 6400 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 00:05:53.293322 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.293303 6400 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 00:05:53.293431 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.293398 6400 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 00:05:53.293494 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.293447 6400 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 00:05:53.293494 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.293454 6400 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 00:05:53.293588 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.293497 6400 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:05:53.293430683Z","UUID":"1735ba7f-4891-4bf4-b99b-ed14c9663a5a","Handler":null,"Name":"","Endpoint":""} Nov 03 00:05:53.295045 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:05:53.294999 6400 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 00:05:53.295133 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:05:53.295067 6400 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-199.us-west-1.compute.internal\" not found" Nov 03 00:05:53.295753 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.295737 6400 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 00:05:53.295808 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.295763 6400 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 00:05:53.330734 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.330701 6400 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:05:53.330880 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.330758 6400 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 00:05:53.330880 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.330761 6400 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:05:53.330880 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.330850 6400 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:05:53.393722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.393687 6400 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:05:53.394657 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.394634 6400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:05:53.394760 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.394662 6400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:05:53.394760 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.394672 6400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-199.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:05:53.394760 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.394719 6400 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.428286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.428268 6400 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:05:53.473229 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.473188 6400 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal"] Nov 03 00:05:53.473805 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.473774 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" event={"ID":"bdf730cee51b9b033223ceb412560cad","Type":"ContainerStarted","Data":"9c282d77a4d35a470acb2347fcd5d16c785952f2371ec8bff55e3dbad13dde96"} Nov 03 00:05:53.473881 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.473808 6400 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.473925 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.473812 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" event={"ID":"bdf730cee51b9b033223ceb412560cad","Type":"ContainerDied","Data":"56df28890ee2e20e4617ba3c42b659c9055b084946299d3a316b187fd8160021"} Nov 03 00:05:53.473961 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.473946 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" event={"ID":"bdf730cee51b9b033223ceb412560cad","Type":"ContainerStarted","Data":"7c1d91f20f7497418646f99c8223b2ad691358891c6cc6c73ae84e841d76e320"} Nov 03 00:05:53.474009 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.473999 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" event={"ID":"cf58af1ed7bc599c71fbbaa4e8065ef6","Type":"ContainerStarted","Data":"76e4943e4878c599f3f55aea2fef2f8b5268bf67a9bb8afaebfd070560289cbd"} Nov 03 00:05:53.474043 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.474014 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" event={"ID":"cf58af1ed7bc599c71fbbaa4e8065ef6","Type":"ContainerStarted","Data":"8ced7c0f225c21078f70aa0d93ae5c09b38e6e4ba40773e95b3e812d45405c33"} Nov 03 00:05:53.474092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.474081 6400 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.502280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.502229 6400 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.502353 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.502286 6400 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.536003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.535977 6400 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:05:53.536077 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:05:53.536016 6400 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.536725 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.536708 6400 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:05:53.536847 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:05:53.536743 6400 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.632404 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.632376 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf58af1ed7bc599c71fbbaa4e8065ef6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"cf58af1ed7bc599c71fbbaa4e8065ef6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.632511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.632414 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bdf730cee51b9b033223ceb412560cad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"bdf730cee51b9b033223ceb412560cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.632511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.632432 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdf730cee51b9b033223ceb412560cad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"bdf730cee51b9b033223ceb412560cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.732738 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.732711 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bdf730cee51b9b033223ceb412560cad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"bdf730cee51b9b033223ceb412560cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.732885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.732781 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bdf730cee51b9b033223ceb412560cad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"bdf730cee51b9b033223ceb412560cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.732885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.732809 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdf730cee51b9b033223ceb412560cad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"bdf730cee51b9b033223ceb412560cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.732885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.732852 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf58af1ed7bc599c71fbbaa4e8065ef6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"cf58af1ed7bc599c71fbbaa4e8065ef6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.732885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.732886 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf58af1ed7bc599c71fbbaa4e8065ef6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"cf58af1ed7bc599c71fbbaa4e8065ef6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:53.733031 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:53.732896 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdf730cee51b9b033223ceb412560cad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" (UID: \"bdf730cee51b9b033223ceb412560cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:54.120658 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.120628 6400 apiserver.go:52] "Watching apiserver" Nov 03 00:05:54.224704 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.224667 6400 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 00:05:54.225370 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.225346 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mzcwb","openshift-monitoring/telemeter-client-5d6548ffc8-8tldc","openshift-cluster-node-tuning-operator/tuned-fnvz5","openshift-dns/node-resolver-8cpwh","openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk","openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9","openshift-network-diagnostics/network-check-target-x6rmn","openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s","openshift-multus/multus-ssc4h","openshift-multus/network-metrics-daemon-67vkk","openshift-insights/insights-runtime-extractor-6v6bq","openshift-monitoring/metrics-server-745647dff6-lq2qf","kube-system/konnectivity-agent-njmnx","kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal","openshift-console/console-c6c8d65bd-fhjhr","openshift-dns/dns-default-rwmxk","openshift-multus/multus-additional-cni-plugins-jkz9t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx","kube-system/global-pull-secret-syncer-52cjz","openshift-ingress-canary/ingress-canary-cdsd8","openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm","openshift-network-operator/iptables-alerter-7z644","openshift-ovn-kubernetes/ovnkube-node-g9lrv","openshift-image-registry/node-ca-nckb2","openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal"] Nov 03 00:05:54.246205 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.246183 6400 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:54.246331 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.246322 6400 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:54.281995 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.281973 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pkcxp\"" Nov 03 00:05:54.281995 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.281987 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f67qw\"" Nov 03 00:05:54.282236 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.281973 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 00:05:54.282236 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.282033 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ptgp6\"" Nov 03 00:05:54.282236 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.282125 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 00:05:54.282449 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.282435 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z6zhf\"" Nov 03 00:05:54.283075 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.283055 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 00:05:54.283177 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.283089 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k6628\"" Nov 03 00:05:54.283177 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.283164 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g7949\"" Nov 03 00:05:54.283541 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.283525 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gd4nl\"" Nov 03 00:05:54.283700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.283681 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-scmmg\"" Nov 03 00:05:54.283775 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.283763 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8glgl\"" Nov 03 00:05:54.284047 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.284034 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 00:05:54.284168 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.284154 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 00:05:54.285011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.284994 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 00:05:54.285076 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285007 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 00:05:54.285076 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285042 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:05:54.285226 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285215 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 00:05:54.285506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285494 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 00:05:54.285567 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285503 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 00:05:54.285567 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285530 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 00:05:54.285567 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285542 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 00:05:54.285567 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285544 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 00:05:54.285567 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285566 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 00:05:54.285765 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285593 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 00:05:54.285765 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285603 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 00:05:54.285915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285897 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 00:05:54.285915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285913 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 00:05:54.286018 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285918 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 00:05:54.286018 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285928 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 00:05:54.286018 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285921 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 00:05:54.286018 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.285900 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 00:05:54.286018 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.286015 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 00:05:54.286202 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.286050 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 00:05:54.286202 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.286113 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fmpf5\"" Nov 03 00:05:54.286202 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.286158 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 00:05:54.286289 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.286261 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 00:05:54.286318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.286311 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 00:05:54.286348 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.286333 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:05:54.286546 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.286534 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 00:05:54.287694 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.287683 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 00:05:54.287745 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.287693 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vqr8l\"" Nov 03 00:05:54.287880 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.287869 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 00:05:54.287944 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.287932 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nhzpn\"" Nov 03 00:05:54.326851 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.326834 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 00:05:54.327808 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.327790 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 00:05:54.327916 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.327894 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 00:05:54.328083 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328062 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 00:05:54.328181 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328086 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 00:05:54.328181 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328096 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 00:05:54.328181 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328112 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 00:05:54.328181 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328126 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 00:05:54.328181 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328131 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 00:05:54.328411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328393 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 00:05:54.328464 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328395 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 00:05:54.328464 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328416 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 00:05:54.328464 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328421 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 00:05:54.328464 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328415 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 00:05:54.328464 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328399 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 00:05:54.328616 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328579 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 00:05:54.329023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.328996 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 00:05:54.329329 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.329311 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jkmg8\"" Nov 03 00:05:54.329559 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.329543 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 00:05:54.329840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.329804 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 00:05:54.329975 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.329957 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bkhng\"" Nov 03 00:05:54.330032 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.329997 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 00:05:54.330269 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.330253 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 00:05:54.332005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.331985 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8pmbc\"" Nov 03 00:05:54.332083 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.332069 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ee34tp9efdls7\"" Nov 03 00:05:54.332130 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.332072 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q5tjj\"" Nov 03 00:05:54.333263 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.333240 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 00:05:54.333263 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.333256 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 00:05:54.333370 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.333361 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 00:05:54.333416 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.333401 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 00:05:54.333458 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.333413 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-d8jlp\"" Nov 03 00:05:54.333505 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.333482 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 00:05:54.333578 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.333254 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 00:05:54.333685 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.333619 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 00:05:54.334025 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.334014 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p8dw8\"" Nov 03 00:05:54.334097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.334029 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 00:05:54.334097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.334048 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 00:05:54.334400 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.334383 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 00:05:54.334439 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.334412 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 00:05:54.334569 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.334554 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bspg5up4v8aee\"" Nov 03 00:05:54.334624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.334562 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nxwbs\"" Nov 03 00:05:54.334624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.334569 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 00:05:54.334624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.334583 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7jrlf\"" Nov 03 00:05:54.334624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.334598 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-sfwn4\"" Nov 03 00:05:54.334763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.334641 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 00:05:54.335304 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335286 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.335368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335324 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df88n\" (UniqueName: \"kubernetes.io/projected/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-kube-api-access-df88n\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.335368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335351 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86519442-21ae-43dc-98bc-730f8212e34b-config-volume\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:54.335368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335354 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 00:05:54.335510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335402 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-znq56\"" Nov 03 00:05:54.335510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335400 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58742d3c-ca4d-424e-aec7-35661094ad41-etc-tuned\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.335510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335439 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 00:05:54.335510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335456 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5367d682-754b-4925-ae57-40a1246d15e2-iptables-alerter-script\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:05:54.335510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335476 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58742d3c-ca4d-424e-aec7-35661094ad41-etc-tuned\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.335510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335486 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9967e995-dc16-4869-bb2f-02eb340eba3e-metrics-client-ca\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.335510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335512 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrpd\" (UniqueName: \"kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd\") pod \"network-check-target-x6rmn\" (UID: \"fd761dae-da21-44e4-adad-287a5d204bd3\") " pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335536 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-oauth-serving-cert\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335560 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgv8\" (UniqueName: \"kubernetes.io/projected/b962e9e5-781e-450c-8c3f-17e217c67df6-kube-api-access-sqgv8\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335575 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335577 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86519442-21ae-43dc-98bc-730f8212e34b-config-volume\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335587 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj8fg\" (UniqueName: \"kubernetes.io/projected/5367d682-754b-4925-ae57-40a1246d15e2-kube-api-access-pj8fg\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335615 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335620 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-system-cni-dir\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335646 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335673 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9zg\" (UniqueName: \"kubernetes.io/projected/423516a7-579e-492a-93dc-0c21e3faa673-kube-api-access-9g9zg\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335686 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335699 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-slash\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335712 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9967e995-dc16-4869-bb2f-02eb340eba3e-metrics-client-ca\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335722 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-sys\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335743 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335746 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddae2ef2-bac4-44c3-bef2-0d40c705c141-hosts-file\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335769 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysconfig\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335789 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-lib-modules\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.335806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335804 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5367d682-754b-4925-ae57-40a1246d15e2-iptables-alerter-script\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335848 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8kfz4\"" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335814 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335516 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335913 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335918 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335941 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhkl\" (UniqueName: \"kubernetes.io/projected/169c66f9-91af-4e46-bf67-019e0cec4a43-kube-api-access-kxhkl\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.335990 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336015 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-serviceca\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336045 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-cni-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336072 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336095 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5h6xs\"" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336125 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-oauth-serving-cert\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336134 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336071 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-serving-cert\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336281 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-serving-cert\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336279 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf82aaa-e340-4a97-a3f4-297694eccab3-original-pull-secret\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336326 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b962e9e5-781e-450c-8c3f-17e217c67df6-data-volume\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336370 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-metrics-certs\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336375 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336443 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58742d3c-ca4d-424e-aec7-35661094ad41-tmp\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336494 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b962e9e5-781e-450c-8c3f-17e217c67df6-data-volume\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336563 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 00:05:54.336585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336564 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-cni-bin\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336593 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58742d3c-ca4d-424e-aec7-35661094ad41-tmp\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336641 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fdf82aaa-e340-4a97-a3f4-297694eccab3-dbus\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336692 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-log-socket\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336804 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b962e9e5-781e-450c-8c3f-17e217c67df6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336845 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-serviceca\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336856 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-cni-multus\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336876 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0-cert\") pod \"ingress-canary-cdsd8\" (UID: \"6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0\") " pod="openshift-ingress-canary/ingress-canary-cdsd8" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.336987 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337018 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-registration-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337029 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b962e9e5-781e-450c-8c3f-17e217c67df6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337043 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-systemd\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337117 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0-cert\") pod \"ingress-canary-cdsd8\" (UID: \"6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0\") " pod="openshift-ingress-canary/ingress-canary-cdsd8" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337158 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxj7\" (UniqueName: \"kubernetes.io/projected/4aa46084-b411-4ee1-98bb-145b0e8438c7-kube-api-access-wfxj7\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337176 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovnkube-config\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337195 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fdf82aaa-e340-4a97-a3f4-297694eccab3-kubelet-config\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337220 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-console-config\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337328 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysctl-d\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.337357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337361 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-cnibin\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337408 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8sg\" (UniqueName: \"kubernetes.io/projected/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-kube-api-access-jg8sg\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337438 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337461 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-socket-dir-parent\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337470 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovnkube-config\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337491 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-k8s-cni-cncf-io\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337567 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337643 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-var-lib-kubelet\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337663 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4aa46084-b411-4ee1-98bb-145b0e8438c7-cni-binary-copy\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337704 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86519442-21ae-43dc-98bc-730f8212e34b-tmp-dir\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337768 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86519442-21ae-43dc-98bc-730f8212e34b-tmp-dir\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337778 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-console-config\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337800 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-socket-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337868 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-sys-fs\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337908 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-multus-certs\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337907 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4aa46084-b411-4ee1-98bb-145b0e8438c7-cni-binary-copy\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.337975 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/423516a7-579e-492a-93dc-0c21e3faa673-tmp\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338026 6400 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338048 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/423516a7-579e-492a-93dc-0c21e3faa673-tmp\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338086 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-systemd-units\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.338096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338109 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-system-cni-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338132 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-cni-netd\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338160 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338239 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-tls\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338256 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-modprobe-d\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338271 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-service-ca\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338285 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-trusted-ca-bundle\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338291 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338301 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7314649e-0fd8-4bcd-bcfa-5140fb2711cf-agent-certs\") pod \"konnectivity-agent-njmnx\" (UID: \"7314649e-0fd8-4bcd-bcfa-5140fb2711cf\") " pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338337 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-run\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338365 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-etc-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338395 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338446 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-service-ca\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338471 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-systemd\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338491 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-ovn\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338502 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-tls\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338531 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-root\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338573 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-textfile\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338589 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338603 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-wtmp\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338614 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7314649e-0fd8-4bcd-bcfa-5140fb2711cf-agent-certs\") pod \"konnectivity-agent-njmnx\" (UID: \"7314649e-0fd8-4bcd-bcfa-5140fb2711cf\") " pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338644 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-textfile\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338648 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovnkube-script-lib\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338673 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/169c66f9-91af-4e46-bf67-019e0cec4a43-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338690 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86519442-21ae-43dc-98bc-730f8212e34b-metrics-tls\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338711 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-host\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:05:54.338763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338749 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lwp\" (UniqueName: \"kubernetes.io/projected/43785dfc-6e27-45e2-8608-5fec9dfa4a86-kube-api-access-28lwp\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338781 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7314649e-0fd8-4bcd-bcfa-5140fb2711cf-konnectivity-ca\") pod \"konnectivity-agent-njmnx\" (UID: \"7314649e-0fd8-4bcd-bcfa-5140fb2711cf\") " pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338807 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-node-log\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338814 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovnkube-script-lib\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338856 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-cni-bin\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338881 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338904 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86519442-21ae-43dc-98bc-730f8212e34b-metrics-tls\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338907 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgv7z\" (UniqueName: \"kubernetes.io/projected/9967e995-dc16-4869-bb2f-02eb340eba3e-kube-api-access-xgv7z\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338938 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-kubernetes\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338960 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-conf-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338983 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5bx\" (UniqueName: \"kubernetes.io/projected/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0-kube-api-access-gp5bx\") pod \"ingress-canary-cdsd8\" (UID: \"6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0\") " pod="openshift-ingress-canary/ingress-canary-cdsd8" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.338999 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/169c66f9-91af-4e46-bf67-019e0cec4a43-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339020 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b962e9e5-781e-450c-8c3f-17e217c67df6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339040 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-kubelet\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339055 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-env-overrides\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339070 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz5nz\" (UniqueName: \"kubernetes.io/projected/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-kube-api-access-tz5nz\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339072 6400 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339085 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-oauth-config\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339101 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339104 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7314649e-0fd8-4bcd-bcfa-5140fb2711cf-konnectivity-ca\") pod \"konnectivity-agent-njmnx\" (UID: \"7314649e-0fd8-4bcd-bcfa-5140fb2711cf\") " pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339119 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-run-netns\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339120 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/169c66f9-91af-4e46-bf67-019e0cec4a43-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339135 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-sys\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339150 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-trusted-ca-bundle\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339163 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-netns\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:05:54.339117 6400 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339275 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cnibin\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339305 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/169c66f9-91af-4e46-bf67-019e0cec4a43-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339332 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovn-node-metrics-cert\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339376 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b962e9e5-781e-450c-8c3f-17e217c67df6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339445 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339446 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-var-lib-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339487 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b962e9e5-781e-450c-8c3f-17e217c67df6-crio-socket\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339528 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/43785dfc-6e27-45e2-8608-5fec9dfa4a86-env-overrides\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339535 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9pj8\" (UniqueName: \"kubernetes.io/projected/ddae2ef2-bac4-44c3-bef2-0d40c705c141-kube-api-access-z9pj8\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339546 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-oauth-config\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339570 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95txv\" (UniqueName: \"kubernetes.io/projected/86519442-21ae-43dc-98bc-730f8212e34b-kube-api-access-95txv\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339576 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/43785dfc-6e27-45e2-8608-5fec9dfa4a86-ovn-node-metrics-cert\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339685 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysctl-conf\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339716 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-hostroot\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339746 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc96d\" (UniqueName: \"kubernetes.io/projected/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-kube-api-access-nc96d\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339776 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5367d682-754b-4925-ae57-40a1246d15e2-host-slash\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:05:54.339779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339808 6400 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339808 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339858 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-device-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:05:54.339863 6400 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-199.us-west-1.compute.internal" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339927 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-etc-kubernetes\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339955 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/169c66f9-91af-4e46-bf67-019e0cec4a43-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.339982 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddae2ef2-bac4-44c3-bef2-0d40c705c141-tmp-dir\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.340002 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-host\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.340038 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.340066 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddae2ef2-bac4-44c3-bef2-0d40c705c141-tmp-dir\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.340131 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clcmb\" (UniqueName: \"kubernetes.io/projected/58742d3c-ca4d-424e-aec7-35661094ad41-kube-api-access-clcmb\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.340162 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/169c66f9-91af-4e46-bf67-019e0cec4a43-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.340180 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-daemon-config\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.340247 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-kubelet\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.341133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.340368 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-daemon-config\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.393535 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.393480 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgv8\" (UniqueName: \"kubernetes.io/projected/b962e9e5-781e-450c-8c3f-17e217c67df6-kube-api-access-sqgv8\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:54.393638 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.393545 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj8fg\" (UniqueName: \"kubernetes.io/projected/5367d682-754b-4925-ae57-40a1246d15e2-kube-api-access-pj8fg\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:05:54.393638 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.393627 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df88n\" (UniqueName: \"kubernetes.io/projected/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-kube-api-access-df88n\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.393739 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.393689 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrpd\" (UniqueName: \"kubernetes.io/projected/fd761dae-da21-44e4-adad-287a5d204bd3-kube-api-access-cxrpd\") pod \"network-check-target-x6rmn\" (UID: \"fd761dae-da21-44e4-adad-287a5d204bd3\") " pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:05:54.393794 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.393778 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9zg\" (UniqueName: \"kubernetes.io/projected/423516a7-579e-492a-93dc-0c21e3faa673-kube-api-access-9g9zg\") pod \"console-c6c8d65bd-fhjhr\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:54.394920 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.394904 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxj7\" (UniqueName: \"kubernetes.io/projected/4aa46084-b411-4ee1-98bb-145b0e8438c7-kube-api-access-wfxj7\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.395277 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.395262 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8sg\" (UniqueName: \"kubernetes.io/projected/ea31369b-b8c9-4405-ab0c-6796cd41cd7a-kube-api-access-jg8sg\") pod \"network-metrics-daemon-67vkk\" (UID: \"ea31369b-b8c9-4405-ab0c-6796cd41cd7a\") " pod="openshift-multus/network-metrics-daemon-67vkk" Nov 03 00:05:54.395553 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.395535 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28lwp\" (UniqueName: \"kubernetes.io/projected/43785dfc-6e27-45e2-8608-5fec9dfa4a86-kube-api-access-28lwp\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.395660 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.395645 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhkl\" (UniqueName: \"kubernetes.io/projected/169c66f9-91af-4e46-bf67-019e0cec4a43-kube-api-access-kxhkl\") pod \"prometheus-operator-9cd89b74c-jm5n9\" (UID: \"169c66f9-91af-4e46-bf67-019e0cec4a43\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jm5n9" Nov 03 00:05:54.396696 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.396666 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clcmb\" (UniqueName: \"kubernetes.io/projected/58742d3c-ca4d-424e-aec7-35661094ad41-kube-api-access-clcmb\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.396696 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.396683 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgv7z\" (UniqueName: \"kubernetes.io/projected/9967e995-dc16-4869-bb2f-02eb340eba3e-kube-api-access-xgv7z\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.396836 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.396774 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95txv\" (UniqueName: \"kubernetes.io/projected/86519442-21ae-43dc-98bc-730f8212e34b-kube-api-access-95txv\") pod \"dns-default-rwmxk\" (UID: \"86519442-21ae-43dc-98bc-730f8212e34b\") " pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:54.397091 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.397073 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz5nz\" (UniqueName: \"kubernetes.io/projected/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-kube-api-access-tz5nz\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.397180 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.397157 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp5bx\" (UniqueName: \"kubernetes.io/projected/6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0-kube-api-access-gp5bx\") pod \"ingress-canary-cdsd8\" (UID: \"6ce1f854-bc96-4ed4-a93a-f7c0092bc4c0\") " pod="openshift-ingress-canary/ingress-canary-cdsd8" Nov 03 00:05:54.397533 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.397513 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9pj8\" (UniqueName: \"kubernetes.io/projected/ddae2ef2-bac4-44c3-bef2-0d40c705c141-kube-api-access-z9pj8\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:05:54.397624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.397611 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc96d\" (UniqueName: \"kubernetes.io/projected/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-kube-api-access-nc96d\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:05:54.440631 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440610 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.440721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440636 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-socket-dir-parent\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.440721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440652 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-k8s-cni-cncf-io\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.440721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440671 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-federate-client-tls\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.440721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440687 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbec544b-d079-48c5-9d21-0e739d7512d4-metrics-client-ca\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.440721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440701 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440735 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440737 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-socket-dir-parent\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440736 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-var-lib-kubelet\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440781 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-socket-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440787 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-var-lib-kubelet\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440813 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-sys-fs\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440854 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-multus-certs\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440858 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-k8s-cni-cncf-io\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440871 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-systemd-units\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440875 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440881 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-socket-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440902 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-system-cni-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440922 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbec544b-d079-48c5-9d21-0e739d7512d4-metrics-client-ca\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440928 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-sys-fs\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440939 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-system-cni-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440929 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-multus-certs\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440903 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-systemd-units\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440939 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-federate-client-tls\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440928 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.440984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.440993 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-tls\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441011 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-cni-netd\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441033 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ba97f37-1485-4215-abe3-3c9083759d8a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441048 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441067 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-cni-netd\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441091 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9ee339b2-b471-4efd-829e-dd247c78685a-metrics-server-audit-profiles\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441115 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-modprobe-d\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441160 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441185 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-run\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441211 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441224 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441231 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-tls\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441233 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441235 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-etc-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441252 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ba97f37-1485-4215-abe3-3c9083759d8a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441278 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-systemd\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441281 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-run\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441305 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ba97f37-1485-4215-abe3-3c9083759d8a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441311 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-systemd\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441305 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-etc-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441330 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-client-ca-bundle\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441345 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-modprobe-d\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441345 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9ee339b2-b471-4efd-829e-dd247c78685a-metrics-server-audit-profiles\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441360 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ba97f37-1485-4215-abe3-3c9083759d8a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441375 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-ovn\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441400 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-root\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441423 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-wtmp\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441435 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441449 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443202b2-a46c-41af-b097-93658ee871f8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vcbjm\" (UID: \"443202b2-a46c-41af-b097-93658ee871f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441458 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-ovn\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441478 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-host\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441499 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-node-exporter-wtmp\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441478 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-root\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441514 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc-host\") pod \"node-ca-nckb2\" (UID: \"4be383eb-5b9b-4b12-b0ee-4dc8a04c02dc\") " pod="openshift-image-registry/node-ca-nckb2" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441501 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-client-ca-bundle\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441506 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ee339b2-b471-4efd-829e-dd247c78685a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441561 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-node-log\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441577 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-cni-bin\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441513 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441603 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441607 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443202b2-a46c-41af-b097-93658ee871f8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vcbjm\" (UID: \"443202b2-a46c-41af-b097-93658ee871f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441618 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-node-log\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441628 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-kubernetes\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441643 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441650 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-conf-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441660 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ee339b2-b471-4efd-829e-dd247c78685a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441670 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441673 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-kubernetes\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441629 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-cni-bin\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441676 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-conf-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441690 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-kubelet\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441715 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441727 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-kubelet\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441732 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-run-netns\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441752 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-sys\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441766 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-netns\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.441748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441781 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cnibin\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441797 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-var-lib-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441845 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b962e9e5-781e-450c-8c3f-17e217c67df6-crio-socket\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441856 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-run-netns\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441858 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-sys\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441868 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-run-netns\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441880 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-cnibin\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441888 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-var-lib-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441861 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441870 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysctl-conf\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441898 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441935 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-hostroot\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441941 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b962e9e5-781e-450c-8c3f-17e217c67df6-crio-socket\") pod \"insights-runtime-extractor-6v6bq\" (UID: \"b962e9e5-781e-450c-8c3f-17e217c67df6\") " pod="openshift-insights/insights-runtime-extractor-6v6bq" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441957 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9mww\" (UniqueName: \"kubernetes.io/projected/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-kube-api-access-c9mww\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441972 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysctl-conf\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441983 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442006 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbec544b-d079-48c5-9d21-0e739d7512d4-tmp\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.441983 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-hostroot\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442030 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl4rl\" (UniqueName: \"kubernetes.io/projected/cbec544b-d079-48c5-9d21-0e739d7512d4-kube-api-access-pl4rl\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442097 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbec544b-d079-48c5-9d21-0e739d7512d4-tmp\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442121 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-secret-metrics-server-client-certs\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442134 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442146 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5367d682-754b-4925-ae57-40a1246d15e2-host-slash\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442184 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5367d682-754b-4925-ae57-40a1246d15e2-host-slash\") pod \"iptables-alerter-7z644\" (UID: \"5367d682-754b-4925-ae57-40a1246d15e2\") " pod="openshift-network-operator/iptables-alerter-7z644" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442189 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-device-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442288 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-etc-kubernetes\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442303 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-device-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442345 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-secret-metrics-server-client-certs\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442376 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-host\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442392 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-etc-kubernetes\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442415 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442438 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-metrics-client-ca\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442428 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-host\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442457 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztn7x\" (UniqueName: \"kubernetes.io/projected/d648f173-aa93-4fe7-999b-b50c88be3440-kube-api-access-ztn7x\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442484 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-kubelet\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442644 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442646 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442688 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442698 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-grpc-tls\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442728 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6lrg\" (UniqueName: \"kubernetes.io/projected/443202b2-a46c-41af-b097-93658ee871f8-kube-api-access-j6lrg\") pod \"monitoring-plugin-77f4f76594-vcbjm\" (UID: \"443202b2-a46c-41af-b097-93658ee871f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442746 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-system-cni-dir\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442688 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-kubelet\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442769 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-slash\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442691 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-metrics-client-ca\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442807 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-host-slash\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442841 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-system-cni-dir\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442849 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-sys\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442869 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddae2ef2-bac4-44c3-bef2-0d40c705c141-hosts-file\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442883 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysconfig\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442897 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-lib-modules\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442948 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddae2ef2-bac4-44c3-bef2-0d40c705c141-hosts-file\") pod \"node-resolver-8cpwh\" (UID: \"ddae2ef2-bac4-44c3-bef2-0d40c705c141\") " pod="openshift-dns/node-resolver-8cpwh" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442980 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cbec544b-d079-48c5-9d21-0e739d7512d4-secret-grpc-tls\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442981 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9967e995-dc16-4869-bb2f-02eb340eba3e-sys\") pod \"node-exporter-mzcwb\" (UID: \"9967e995-dc16-4869-bb2f-02eb340eba3e\") " pod="openshift-monitoring/node-exporter-mzcwb" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442998 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-lib-modules\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.442999 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8664\" (UniqueName: \"kubernetes.io/projected/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-api-access-l8664\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443037 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-cni-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443063 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443084 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-multus-cni-dir\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443090 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443038 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysconfig\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443125 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443143 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-secret-telemeter-client\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443167 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443183 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-cni-bin\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443218 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fdf82aaa-e340-4a97-a3f4-297694eccab3-dbus\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443229 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-cni-bin\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443251 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9ee339b2-b471-4efd-829e-dd247c78685a-audit-log\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443259 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d41c02-03bc-4840-ae2f-a1590dda4ee9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jkz9t\" (UID: \"d5d41c02-03bc-4840-ae2f-a1590dda4ee9\") " pod="openshift-multus/multus-additional-cni-plugins-jkz9t" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443272 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443277 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7qzc\" (UniqueName: \"kubernetes.io/projected/9ee339b2-b471-4efd-829e-dd247c78685a-kube-api-access-x7qzc\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443282 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443304 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-log-socket\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443321 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-cni-multus\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443323 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fdf82aaa-e340-4a97-a3f4-297694eccab3-dbus\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443336 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443338 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-log-socket\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443356 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-registration-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443362 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-host-var-lib-cni-multus\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443368 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443355 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9ee339b2-b471-4efd-829e-dd247c78685a-audit-log\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443380 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-systemd\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443381 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/43785dfc-6e27-45e2-8608-5fec9dfa4a86-run-openvswitch\") pod \"ovnkube-node-g9lrv\" (UID: \"43785dfc-6e27-45e2-8608-5fec9dfa4a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443402 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35dc422a-ae29-4b68-b3fc-efd9b44fa2dd-registration-dir\") pod \"aws-ebs-csi-driver-node-6dwqx\" (UID: \"35dc422a-ae29-4b68-b3fc-efd9b44fa2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dwqx" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443412 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-serving-certs-ca-bundle\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443415 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-secret-telemeter-client\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443449 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-systemd\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443546 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fdf82aaa-e340-4a97-a3f4-297694eccab3-kubelet-config\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443567 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fdf82aaa-e340-4a97-a3f4-297694eccab3-kubelet-config\") pod \"global-pull-secret-syncer-52cjz\" (UID: \"fdf82aaa-e340-4a97-a3f4-297694eccab3\") " pod="kube-system/global-pull-secret-syncer-52cjz" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443571 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysctl-d\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443591 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-cnibin\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443565 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d648f173-aa93-4fe7-999b-b50c88be3440-serving-certs-ca-bundle\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443607 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443626 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-telemeter-client-tls\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443644 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-secret-metrics-server-tls\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443649 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58742d3c-ca4d-424e-aec7-35661094ad41-etc-sysctl-d\") pod \"tuned-fnvz5\" (UID: \"58742d3c-ca4d-424e-aec7-35661094ad41\") " pod="openshift-cluster-node-tuning-operator/tuned-fnvz5" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443649 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4aa46084-b411-4ee1-98bb-145b0e8438c7-cnibin\") pod \"multus-ssc4h\" (UID: \"4aa46084-b411-4ee1-98bb-145b0e8438c7\") " pod="openshift-multus/multus-ssc4h" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443746 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443870 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d648f173-aa93-4fe7-999b-b50c88be3440-telemeter-client-tls\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.443921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.443892 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9ee339b2-b471-4efd-829e-dd247c78685a-secret-metrics-server-tls\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.497467 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.497445 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6lrg\" (UniqueName: \"kubernetes.io/projected/443202b2-a46c-41af-b097-93658ee871f8-kube-api-access-j6lrg\") pod \"monitoring-plugin-77f4f76594-vcbjm\" (UID: \"443202b2-a46c-41af-b097-93658ee871f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:05:54.497609 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.497576 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl4rl\" (UniqueName: \"kubernetes.io/projected/cbec544b-d079-48c5-9d21-0e739d7512d4-kube-api-access-pl4rl\") pod \"thanos-querier-869bd4cbfb-2qs6h\" (UID: \"cbec544b-d079-48c5-9d21-0e739d7512d4\") " pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:54.497700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.497633 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9mww\" (UniqueName: \"kubernetes.io/projected/54b0b5df-4723-4f99-98a0-b7a386f3d0c5-kube-api-access-c9mww\") pod \"openshift-state-metrics-7bd7489957-2psfk\" (UID: \"54b0b5df-4723-4f99-98a0-b7a386f3d0c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2psfk" Nov 03 00:05:54.498726 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.498706 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztn7x\" (UniqueName: \"kubernetes.io/projected/d648f173-aa93-4fe7-999b-b50c88be3440-kube-api-access-ztn7x\") pod \"telemeter-client-5d6548ffc8-8tldc\" (UID: \"d648f173-aa93-4fe7-999b-b50c88be3440\") " pod="openshift-monitoring/telemeter-client-5d6548ffc8-8tldc" Nov 03 00:05:54.498774 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.498758 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7qzc\" (UniqueName: \"kubernetes.io/projected/9ee339b2-b471-4efd-829e-dd247c78685a-kube-api-access-x7qzc\") pod \"metrics-server-745647dff6-lq2qf\" (UID: \"9ee339b2-b471-4efd-829e-dd247c78685a\") " pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:05:54.499028 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.499014 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8664\" (UniqueName: \"kubernetes.io/projected/8ba97f37-1485-4215-abe3-3c9083759d8a-kube-api-access-l8664\") pod \"kube-state-metrics-5dc8d7d89b-7p54s\" (UID: \"8ba97f37-1485-4215-abe3-3c9083759d8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7p54s" Nov 03 00:05:54.598512 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:54.598486 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:55.859707 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:55.859662 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:56.034067 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:56.034037 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:56.779868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:56.779838 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:56.787794 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:56.786624 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-869bd4cbfb-2qs6h" Nov 03 00:05:56.934103 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:56.934071 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rwmxk" Nov 03 00:05:57.150970 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:05:57.150935 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:05:57.468200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:57.468126 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:05:57.810545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:57.810517 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-85584b8b77-dcqq8"] Nov 03 00:05:57.889991 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:57.889961 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:57.893703 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:57.893675 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85584b8b77-dcqq8"] Nov 03 00:05:57.974303 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:57.974275 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-oauth-config\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:57.974648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:57.974318 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-console-config\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:57.974648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:57.974335 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-trusted-ca-bundle\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:57.974648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:57.974441 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-serving-cert\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:57.974648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:57.974469 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/463f10a4-6532-431c-b7a5-67472c923855-tmp\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:57.974648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:57.974487 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-service-ca\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:57.974648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:57.974507 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7ncn\" (UniqueName: \"kubernetes.io/projected/463f10a4-6532-431c-b7a5-67472c923855-kube-api-access-f7ncn\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:57.974648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:57.974606 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-oauth-serving-cert\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:57.994271 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:57.994232 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:58.075543 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.075471 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-serving-cert\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.075543 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.075510 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/463f10a4-6532-431c-b7a5-67472c923855-tmp\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.075543 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.075531 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-service-ca\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.075803 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.075550 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7ncn\" (UniqueName: \"kubernetes.io/projected/463f10a4-6532-431c-b7a5-67472c923855-kube-api-access-f7ncn\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.075803 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.075684 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-oauth-serving-cert\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.075803 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.075741 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-oauth-config\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.075803 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.075799 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-console-config\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.076029 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.075846 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-trusted-ca-bundle\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.076029 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.075968 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/463f10a4-6532-431c-b7a5-67472c923855-tmp\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.076029 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.075996 6400 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 00:05:58.076506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.076412 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-service-ca\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.076506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.076489 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-oauth-serving-cert\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.076506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.076560 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-console-config\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.076865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.076800 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-trusted-ca-bundle\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.078399 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.078371 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-oauth-config\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.078486 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.078444 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-serving-cert\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.133059 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.133033 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7ncn\" (UniqueName: \"kubernetes.io/projected/463f10a4-6532-431c-b7a5-67472c923855-kube-api-access-f7ncn\") pod \"console-85584b8b77-dcqq8\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.189117 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.189091 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:58.222948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.222927 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:05:58.447077 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.446975 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:05:58.447258 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.447242 6400 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:05:58.620569 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:58.620536 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85584b8b77-dcqq8"] Nov 03 00:05:58.623956 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:05:58.623929 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463f10a4_6532_431c_b7a5_67472c923855.slice/crio-cb5818c26f420d2057f48a46b5d61708de2185963cfe20c9f82215c108566f0a WatchSource:0}: Error finding container cb5818c26f420d2057f48a46b5d61708de2185963cfe20c9f82215c108566f0a: Status 404 returned error can't find the container with id cb5818c26f420d2057f48a46b5d61708de2185963cfe20c9f82215c108566f0a Nov 03 00:05:59.266420 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:59.266376 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85584b8b77-dcqq8" event={"ID":"463f10a4-6532-431c-b7a5-67472c923855","Type":"ContainerStarted","Data":"ddcbebadc7be9ef3a0f1dc2abe141b6f2e97f7b75cacf674ed836d9b9585f5d1"} Nov 03 00:05:59.266888 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:59.266438 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85584b8b77-dcqq8" event={"ID":"463f10a4-6532-431c-b7a5-67472c923855","Type":"ContainerStarted","Data":"cb5818c26f420d2057f48a46b5d61708de2185963cfe20c9f82215c108566f0a"} Nov 03 00:05:59.381713 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:59.381658 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-85584b8b77-dcqq8" podStartSLOduration=2.381643798 podStartE2EDuration="2.381643798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:59.381337868 +0000 UTC m=+6.332278037" watchObservedRunningTime="2026-11-03 00:05:59.381643798 +0000 UTC m=+6.332584023" Nov 03 00:05:59.841231 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:05:59.841193 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:00.015142 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:00.015110 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:01.208669 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:01.208634 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:06:01.237250 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:01.237218 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:06:01.241942 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:01.241920 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vcbjm" Nov 03 00:06:02.070054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:02.070026 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9lrv" Nov 03 00:06:02.353004 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:02.352916 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:06:02.391412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:02.391378 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:02.541348 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:02.541318 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:02.939361 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:02.939328 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:06:04.284589 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:04.284555 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:04.304084 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:04.304063 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:06:04.308895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:04.308875 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:06:04.450185 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:04.450159 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:05.292535 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:05.292511 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:06:05.357632 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:05.357605 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:06:05.457490 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:05.457461 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c6c8d65bd-fhjhr"] Nov 03 00:06:06.093631 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:06.093598 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-njmnx" Nov 03 00:06:20.755446 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:20.755400 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:20.910323 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:20.910281 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:21.212139 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:21.212109 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x6rmn" Nov 03 00:06:25.362715 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:25.362688 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:06:25.366679 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:25.366655 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-745647dff6-lq2qf" Nov 03 00:06:27.152449 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:06:27.152409 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:06:29.198236 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:29.198202 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:29.403513 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:29.403478 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:29.905584 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:29.905548 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:30.051705 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.051673 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:30.477763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.477723 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c6c8d65bd-fhjhr" podUID="423516a7-579e-492a-93dc-0c21e3faa673" containerName="console" containerID="cri-o://c8efae2f0a737e313a0f05a730381acee05cb70f622b64db4bd098f32871d9d8" gracePeriod=15 Nov 03 00:06:30.724284 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.724261 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:06:30.859943 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.859908 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423516a7-579e-492a-93dc-0c21e3faa673-tmp\" (UniqueName: \"kubernetes.io/empty-dir/423516a7-579e-492a-93dc-0c21e3faa673-tmp\") pod \"423516a7-579e-492a-93dc-0c21e3faa673\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " Nov 03 00:06:30.860138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.859979 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-trusted-ca-bundle\") pod \"423516a7-579e-492a-93dc-0c21e3faa673\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " Nov 03 00:06:30.860138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860009 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-oauth-serving-cert\") pod \"423516a7-579e-492a-93dc-0c21e3faa673\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " Nov 03 00:06:30.860138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860049 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423516a7-579e-492a-93dc-0c21e3faa673-kube-api-access-9g9zg\" (UniqueName: \"kubernetes.io/projected/423516a7-579e-492a-93dc-0c21e3faa673-kube-api-access-9g9zg\") pod \"423516a7-579e-492a-93dc-0c21e3faa673\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " Nov 03 00:06:30.860308 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860267 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-oauth-config\") pod \"423516a7-579e-492a-93dc-0c21e3faa673\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " Nov 03 00:06:30.860308 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860247 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423516a7-579e-492a-93dc-0c21e3faa673-tmp" pod "423516a7-579e-492a-93dc-0c21e3faa673" (UID: "423516a7-579e-492a-93dc-0c21e3faa673"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:30.860419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860333 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-console-config\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-console-config\") pod \"423516a7-579e-492a-93dc-0c21e3faa673\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " Nov 03 00:06:30.860475 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860415 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-oauth-serving-cert" pod "423516a7-579e-492a-93dc-0c21e3faa673" (UID: "423516a7-579e-492a-93dc-0c21e3faa673"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:30.860475 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860422 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-serving-cert\") pod \"423516a7-579e-492a-93dc-0c21e3faa673\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " Nov 03 00:06:30.860475 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860454 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-trusted-ca-bundle" pod "423516a7-579e-492a-93dc-0c21e3faa673" (UID: "423516a7-579e-492a-93dc-0c21e3faa673"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:30.860627 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860489 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-service-ca\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-service-ca\") pod \"423516a7-579e-492a-93dc-0c21e3faa673\" (UID: \"423516a7-579e-492a-93dc-0c21e3faa673\") " Nov 03 00:06:30.860711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860683 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-console-config" pod "423516a7-579e-492a-93dc-0c21e3faa673" (UID: "423516a7-579e-492a-93dc-0c21e3faa673"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:30.860900 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860874 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-service-ca" pod "423516a7-579e-492a-93dc-0c21e3faa673" (UID: "423516a7-579e-492a-93dc-0c21e3faa673"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:30.861158 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860938 6400 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-console-config\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:30.861158 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860956 6400 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:30.861158 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860972 6400 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/423516a7-579e-492a-93dc-0c21e3faa673-tmp\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:30.861158 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.860987 6400 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-trusted-ca-bundle\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:30.861158 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.861001 6400 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/423516a7-579e-492a-93dc-0c21e3faa673-oauth-serving-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:30.862754 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.862724 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-oauth-config" pod "423516a7-579e-492a-93dc-0c21e3faa673" (UID: "423516a7-579e-492a-93dc-0c21e3faa673"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:30.862872 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.862770 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423516a7-579e-492a-93dc-0c21e3faa673-kube-api-access-9g9zg" pod "423516a7-579e-492a-93dc-0c21e3faa673" (UID: "423516a7-579e-492a-93dc-0c21e3faa673"). InnerVolumeSpecName "kube-api-access-9g9zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:30.862872 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.862786 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-serving-cert" pod "423516a7-579e-492a-93dc-0c21e3faa673" (UID: "423516a7-579e-492a-93dc-0c21e3faa673"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:30.962278 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.962236 6400 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-serving-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:30.962278 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.962270 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g9zg\" (UniqueName: \"kubernetes.io/projected/423516a7-579e-492a-93dc-0c21e3faa673-kube-api-access-9g9zg\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:30.962278 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:30.962280 6400 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/423516a7-579e-492a-93dc-0c21e3faa673-console-oauth-config\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:31.210051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.210019 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b6c7dd6d7-lb2pw"] Nov 03 00:06:31.210336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.210324 6400 cpu_manager.go:414] "containerMap: removing container" podUID="423516a7-579e-492a-93dc-0c21e3faa673" containerName="console" Nov 03 00:06:31.210385 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.210337 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423516a7-579e-492a-93dc-0c21e3faa673" containerName="console" Nov 03 00:06:31.210418 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.210385 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="423516a7-579e-492a-93dc-0c21e3faa673" containerName="console" Nov 03 00:06:31.212301 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.212285 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.312791 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.312763 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6c7dd6d7-lb2pw"] Nov 03 00:06:31.364966 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.364934 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcfdde4-52f4-443a-945b-418781368263-service-ca\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.365146 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.364974 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcfdde4-52f4-443a-945b-418781368263-console-serving-cert\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.365146 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.365010 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcfdde4-52f4-443a-945b-418781368263-oauth-serving-cert\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.365146 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.365081 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dcfdde4-52f4-443a-945b-418781368263-tmp\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.365246 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.365151 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zwmp\" (UniqueName: \"kubernetes.io/projected/2dcfdde4-52f4-443a-945b-418781368263-kube-api-access-2zwmp\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.365246 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.365176 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcfdde4-52f4-443a-945b-418781368263-console-oauth-config\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.365246 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.365196 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcfdde4-52f4-443a-945b-418781368263-trusted-ca-bundle\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.365246 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.365216 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcfdde4-52f4-443a-945b-418781368263-console-config\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.382992 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.382953 6400 generic.go:356] "Generic (PLEG): container finished" podID="423516a7-579e-492a-93dc-0c21e3faa673" containerID="c8efae2f0a737e313a0f05a730381acee05cb70f622b64db4bd098f32871d9d8" exitCode=0 Nov 03 00:06:31.383162 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.383025 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6c8d65bd-fhjhr" event={"ID":"423516a7-579e-492a-93dc-0c21e3faa673","Type":"ContainerDied","Data":"c8efae2f0a737e313a0f05a730381acee05cb70f622b64db4bd098f32871d9d8"} Nov 03 00:06:31.383162 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.383033 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6c8d65bd-fhjhr" Nov 03 00:06:31.383162 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.383092 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6c8d65bd-fhjhr" event={"ID":"423516a7-579e-492a-93dc-0c21e3faa673","Type":"ContainerDied","Data":"806034f7a24d9e9dd18667d2c50a1a57131ab581d5b4d0bb53b4b3629a468fc7"} Nov 03 00:06:31.383162 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.383115 6400 scope.go:122] "RemoveContainer" containerID="c8efae2f0a737e313a0f05a730381acee05cb70f622b64db4bd098f32871d9d8" Nov 03 00:06:31.391188 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.391167 6400 scope.go:122] "RemoveContainer" containerID="c8efae2f0a737e313a0f05a730381acee05cb70f622b64db4bd098f32871d9d8" Nov 03 00:06:31.391445 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:06:31.391424 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8efae2f0a737e313a0f05a730381acee05cb70f622b64db4bd098f32871d9d8\": container with ID starting with c8efae2f0a737e313a0f05a730381acee05cb70f622b64db4bd098f32871d9d8 not found: ID does not exist" containerID="c8efae2f0a737e313a0f05a730381acee05cb70f622b64db4bd098f32871d9d8" Nov 03 00:06:31.391501 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.391460 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8efae2f0a737e313a0f05a730381acee05cb70f622b64db4bd098f32871d9d8"} err="failed to get container status \"c8efae2f0a737e313a0f05a730381acee05cb70f622b64db4bd098f32871d9d8\": rpc error: code = NotFound desc = could not find container \"c8efae2f0a737e313a0f05a730381acee05cb70f622b64db4bd098f32871d9d8\": container with ID starting with c8efae2f0a737e313a0f05a730381acee05cb70f622b64db4bd098f32871d9d8 not found: ID does not exist" Nov 03 00:06:31.466268 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.466187 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcfdde4-52f4-443a-945b-418781368263-oauth-serving-cert\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.466268 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.466228 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dcfdde4-52f4-443a-945b-418781368263-tmp\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.466450 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.466352 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwmp\" (UniqueName: \"kubernetes.io/projected/2dcfdde4-52f4-443a-945b-418781368263-kube-api-access-2zwmp\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.466450 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.466386 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcfdde4-52f4-443a-945b-418781368263-console-oauth-config\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.466450 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.466424 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcfdde4-52f4-443a-945b-418781368263-trusted-ca-bundle\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.466595 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.466459 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcfdde4-52f4-443a-945b-418781368263-console-config\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.466595 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.466496 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcfdde4-52f4-443a-945b-418781368263-service-ca\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.466688 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.466638 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcfdde4-52f4-443a-945b-418781368263-console-serving-cert\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.466734 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.466696 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dcfdde4-52f4-443a-945b-418781368263-tmp\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.467089 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.467065 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcfdde4-52f4-443a-945b-418781368263-oauth-serving-cert\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.467218 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.467197 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcfdde4-52f4-443a-945b-418781368263-service-ca\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.467336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.467317 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcfdde4-52f4-443a-945b-418781368263-console-config\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.467443 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.467422 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcfdde4-52f4-443a-945b-418781368263-trusted-ca-bundle\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.469125 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.469109 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcfdde4-52f4-443a-945b-418781368263-console-serving-cert\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.469181 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.469147 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcfdde4-52f4-443a-945b-418781368263-console-oauth-config\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.518796 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.518762 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:31.522255 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.522236 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zwmp\" (UniqueName: \"kubernetes.io/projected/2dcfdde4-52f4-443a-945b-418781368263-kube-api-access-2zwmp\") pod \"console-5b6c7dd6d7-lb2pw\" (UID: \"2dcfdde4-52f4-443a-945b-418781368263\") " pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.543664 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.543635 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c6c8d65bd-fhjhr"] Nov 03 00:06:31.549532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.549509 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c6c8d65bd-fhjhr"] Nov 03 00:06:31.728081 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.727987 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:31.822300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.822265 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:31.902684 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:31.902653 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:32.063425 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:32.063390 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:32.191542 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:32.191509 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6c7dd6d7-lb2pw"] Nov 03 00:06:32.194523 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:06:32.194498 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dcfdde4_52f4_443a_945b_418781368263.slice/crio-42a9f59cdb87467683d2f46b1ab9c31793728203dc337840a7002ba87e8e5924 WatchSource:0}: Error finding container 42a9f59cdb87467683d2f46b1ab9c31793728203dc337840a7002ba87e8e5924: Status 404 returned error can't find the container with id 42a9f59cdb87467683d2f46b1ab9c31793728203dc337840a7002ba87e8e5924 Nov 03 00:06:32.388734 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:32.388639 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6c7dd6d7-lb2pw" event={"ID":"2dcfdde4-52f4-443a-945b-418781368263","Type":"ContainerStarted","Data":"f20b6bca1be0805a84e003bbd9d27362e20ce0f07364d95d6585766ca8f82b4b"} Nov 03 00:06:32.388734 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:32.388704 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6c7dd6d7-lb2pw" event={"ID":"2dcfdde4-52f4-443a-945b-418781368263","Type":"ContainerStarted","Data":"42a9f59cdb87467683d2f46b1ab9c31793728203dc337840a7002ba87e8e5924"} Nov 03 00:06:32.501619 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:32.501519 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b6c7dd6d7-lb2pw" podStartSLOduration=1.5015017099999999 podStartE2EDuration="1.50150171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:06:32.501239429 +0000 UTC m=+39.452179604" watchObservedRunningTime="2026-11-03 00:06:32.50150171 +0000 UTC m=+39.452441868" Nov 03 00:06:33.104522 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:33.104484 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:33.176349 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:33.176313 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423516a7-579e-492a-93dc-0c21e3faa673" path="/var/lib/kubelet/pods/423516a7-579e-492a-93dc-0c21e3faa673/volumes" Nov 03 00:06:33.349053 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:33.349025 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:39.477269 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.477236 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 00:06:39.480412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.480396 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.531939 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.531898 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.531939 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.531939 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.532151 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.531963 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/990360fa-7788-4af4-8d22-47b4452f6a7c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.532151 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532066 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-web-config\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.532151 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532089 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/990360fa-7788-4af4-8d22-47b4452f6a7c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.532151 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532106 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.532151 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532122 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.532151 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532144 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-config-volume\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.532329 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532260 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/990360fa-7788-4af4-8d22-47b4452f6a7c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.532329 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532297 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/990360fa-7788-4af4-8d22-47b4452f6a7c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.532329 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532324 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.532470 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532451 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/990360fa-7788-4af4-8d22-47b4452f6a7c-config-out\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.532580 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532492 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjjl6\" (UniqueName: \"kubernetes.io/projected/990360fa-7788-4af4-8d22-47b4452f6a7c-kube-api-access-fjjl6\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.532580 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532539 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 00:06:39.532580 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532570 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 00:06:39.532580 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532582 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 00:06:39.532894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532683 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 00:06:39.532894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532749 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 00:06:39.532894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532753 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 00:06:39.532894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532755 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 00:06:39.532894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532800 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 00:06:39.532894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.532872 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k72nn\"" Nov 03 00:06:39.583642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.583609 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 00:06:39.632970 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.632936 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.632974 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/990360fa-7788-4af4-8d22-47b4452f6a7c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.633005 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-web-config\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.633039 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/990360fa-7788-4af4-8d22-47b4452f6a7c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.633123 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.633150 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.633178 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-config-volume\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.633227 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/990360fa-7788-4af4-8d22-47b4452f6a7c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.633266 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/990360fa-7788-4af4-8d22-47b4452f6a7c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.633295 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.633328 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/990360fa-7788-4af4-8d22-47b4452f6a7c-config-out\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.633364 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjjl6\" (UniqueName: \"kubernetes.io/projected/990360fa-7788-4af4-8d22-47b4452f6a7c-kube-api-access-fjjl6\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.633400 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633694 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.633413 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/990360fa-7788-4af4-8d22-47b4452f6a7c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.633998 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.633972 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/990360fa-7788-4af4-8d22-47b4452f6a7c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.636507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.636477 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-web-config\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.636621 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.636477 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.636681 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.636655 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.636939 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.636889 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-config-volume\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.637053 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.637014 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/990360fa-7788-4af4-8d22-47b4452f6a7c-config-out\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.637283 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.637086 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.637283 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.637205 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.638581 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.638560 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/990360fa-7788-4af4-8d22-47b4452f6a7c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.639163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.639142 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/990360fa-7788-4af4-8d22-47b4452f6a7c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.663570 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.663546 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 00:06:39.674384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.674351 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/990360fa-7788-4af4-8d22-47b4452f6a7c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.692434 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.692409 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjjl6\" (UniqueName: \"kubernetes.io/projected/990360fa-7788-4af4-8d22-47b4452f6a7c-kube-api-access-fjjl6\") pod \"alertmanager-main-0\" (UID: \"990360fa-7788-4af4-8d22-47b4452f6a7c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:39.797102 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:39.797073 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 00:06:40.219880 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:40.219780 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 00:06:40.223245 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:06:40.223218 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990360fa_7788_4af4_8d22_47b4452f6a7c.slice/crio-c1a48ac468f0a755c8b84c8f561c3ca4866963b02be41083b8b0a8cb5486cd19 WatchSource:0}: Error finding container c1a48ac468f0a755c8b84c8f561c3ca4866963b02be41083b8b0a8cb5486cd19: Status 404 returned error can't find the container with id c1a48ac468f0a755c8b84c8f561c3ca4866963b02be41083b8b0a8cb5486cd19 Nov 03 00:06:40.423922 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:40.423889 6400 generic.go:356] "Generic (PLEG): container finished" podID="990360fa-7788-4af4-8d22-47b4452f6a7c" containerID="7e2f98233ff25af16d5195d47e21e5e183cde60cdd707ba70c69f0e167b7a860" exitCode=0 Nov 03 00:06:40.424105 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:40.423928 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990360fa-7788-4af4-8d22-47b4452f6a7c","Type":"ContainerDied","Data":"7e2f98233ff25af16d5195d47e21e5e183cde60cdd707ba70c69f0e167b7a860"} Nov 03 00:06:40.424105 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:40.424010 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990360fa-7788-4af4-8d22-47b4452f6a7c","Type":"ContainerStarted","Data":"c1a48ac468f0a755c8b84c8f561c3ca4866963b02be41083b8b0a8cb5486cd19"} Nov 03 00:06:40.425349 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:40.425330 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:06:41.823120 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:41.823084 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:41.823607 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:41.823405 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:41.828721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:41.828508 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:42.434931 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:42.434896 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990360fa-7788-4af4-8d22-47b4452f6a7c","Type":"ContainerStarted","Data":"0499f544608df5be98aaff2f986805f0d6d8aea1a5a8326df956191a032ad516"} Nov 03 00:06:42.439968 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:42.439944 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b6c7dd6d7-lb2pw" Nov 03 00:06:42.583584 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:42.583199 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85584b8b77-dcqq8"] Nov 03 00:06:43.443298 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:43.443256 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990360fa-7788-4af4-8d22-47b4452f6a7c","Type":"ContainerStarted","Data":"b8990f45d8cb503b40d2bdb3109f903f75a106ae82a1d6e083e19adfa6cb2f48"} Nov 03 00:06:43.443763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:43.443347 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990360fa-7788-4af4-8d22-47b4452f6a7c","Type":"ContainerStarted","Data":"7fd95fce27185f2db793aca91ef1bc5c70c5d2765bc4212b5ef12d060ce919fb"} Nov 03 00:06:43.443763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:43.443393 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990360fa-7788-4af4-8d22-47b4452f6a7c","Type":"ContainerStarted","Data":"e599f5866f51e6f25af30401006c039abc57f99759a2849cca01859ed289ed79"} Nov 03 00:06:43.443763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:43.443431 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990360fa-7788-4af4-8d22-47b4452f6a7c","Type":"ContainerStarted","Data":"b499e5694b58d35a12548ea81d2634a7eb81b217716d9d469e48d835da1cf8d9"} Nov 03 00:06:43.443763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:43.443468 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"990360fa-7788-4af4-8d22-47b4452f6a7c","Type":"ContainerStarted","Data":"fecccf7c9bffea2b880a3996c49354bce27369574a17a08b6ad01c09ccaa50a6"} Nov 03 00:06:43.568201 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:43.568148 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.6447088709999997 podStartE2EDuration="4.568133475s" totalImagesPullingTime="1.911381432s" totalInitContainerRuntime="12.043172ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:06:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:06:43.566944218 +0000 UTC m=+50.517884378" watchObservedRunningTime="2026-11-03 00:06:43.568133475 +0000 UTC m=+50.519073633" Nov 03 00:06:44.199399 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:44.199360 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:44.403240 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:44.403194 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:57.151495 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:06:57.151452 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:06:58.336439 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:06:58.336401 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:07.605516 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.605477 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-85584b8b77-dcqq8" podUID="463f10a4-6532-431c-b7a5-67472c923855" containerName="console" containerID="cri-o://ddcbebadc7be9ef3a0f1dc2abe141b6f2e97f7b75cacf674ed836d9b9585f5d1" gracePeriod=15 Nov 03 00:07:07.857864 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.857790 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:07:07.985294 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985258 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-oauth-config\") pod \"463f10a4-6532-431c-b7a5-67472c923855\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " Nov 03 00:07:07.985475 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985302 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463f10a4-6532-431c-b7a5-67472c923855-tmp\" (UniqueName: \"kubernetes.io/empty-dir/463f10a4-6532-431c-b7a5-67472c923855-tmp\") pod \"463f10a4-6532-431c-b7a5-67472c923855\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " Nov 03 00:07:07.985475 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985340 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-oauth-serving-cert\") pod \"463f10a4-6532-431c-b7a5-67472c923855\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " Nov 03 00:07:07.985475 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985394 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-console-config\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-console-config\") pod \"463f10a4-6532-431c-b7a5-67472c923855\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " Nov 03 00:07:07.985475 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985456 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-service-ca\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-service-ca\") pod \"463f10a4-6532-431c-b7a5-67472c923855\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " Nov 03 00:07:07.985676 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985497 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-trusted-ca-bundle\") pod \"463f10a4-6532-431c-b7a5-67472c923855\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " Nov 03 00:07:07.985676 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985541 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-serving-cert\") pod \"463f10a4-6532-431c-b7a5-67472c923855\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " Nov 03 00:07:07.985676 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985597 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463f10a4-6532-431c-b7a5-67472c923855-kube-api-access-f7ncn\" (UniqueName: \"kubernetes.io/projected/463f10a4-6532-431c-b7a5-67472c923855-kube-api-access-f7ncn\") pod \"463f10a4-6532-431c-b7a5-67472c923855\" (UID: \"463f10a4-6532-431c-b7a5-67472c923855\") " Nov 03 00:07:07.985676 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985615 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463f10a4-6532-431c-b7a5-67472c923855-tmp" pod "463f10a4-6532-431c-b7a5-67472c923855" (UID: "463f10a4-6532-431c-b7a5-67472c923855"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:07.985889 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985872 6400 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/463f10a4-6532-431c-b7a5-67472c923855-tmp\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:07.985949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985910 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-oauth-serving-cert" pod "463f10a4-6532-431c-b7a5-67472c923855" (UID: "463f10a4-6532-431c-b7a5-67472c923855"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:07.986005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985954 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-console-config" pod "463f10a4-6532-431c-b7a5-67472c923855" (UID: "463f10a4-6532-431c-b7a5-67472c923855"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:07.986005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985960 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-service-ca" pod "463f10a4-6532-431c-b7a5-67472c923855" (UID: "463f10a4-6532-431c-b7a5-67472c923855"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:07.986005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.985977 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-trusted-ca-bundle" pod "463f10a4-6532-431c-b7a5-67472c923855" (UID: "463f10a4-6532-431c-b7a5-67472c923855"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:07.987972 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.987943 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-serving-cert" pod "463f10a4-6532-431c-b7a5-67472c923855" (UID: "463f10a4-6532-431c-b7a5-67472c923855"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:07:07.988201 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.988181 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463f10a4-6532-431c-b7a5-67472c923855-kube-api-access-f7ncn" pod "463f10a4-6532-431c-b7a5-67472c923855" (UID: "463f10a4-6532-431c-b7a5-67472c923855"). InnerVolumeSpecName "kube-api-access-f7ncn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:07.988379 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:07.988345 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-oauth-config" pod "463f10a4-6532-431c-b7a5-67472c923855" (UID: "463f10a4-6532-431c-b7a5-67472c923855"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:07:08.086971 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.086938 6400 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-oauth-serving-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:08.086971 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.086968 6400 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-console-config\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:08.086971 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.086979 6400 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:08.087190 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.086988 6400 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463f10a4-6532-431c-b7a5-67472c923855-trusted-ca-bundle\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:08.087190 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.086999 6400 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-serving-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:08.087190 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.087007 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7ncn\" (UniqueName: \"kubernetes.io/projected/463f10a4-6532-431c-b7a5-67472c923855-kube-api-access-f7ncn\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:08.087190 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.087017 6400 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/463f10a4-6532-431c-b7a5-67472c923855-console-oauth-config\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:08.271352 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.271244 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:08.535612 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.535580 6400 generic.go:356] "Generic (PLEG): container finished" podID="463f10a4-6532-431c-b7a5-67472c923855" containerID="ddcbebadc7be9ef3a0f1dc2abe141b6f2e97f7b75cacf674ed836d9b9585f5d1" exitCode=0 Nov 03 00:07:08.535800 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.535623 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85584b8b77-dcqq8" Nov 03 00:07:08.535800 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.535658 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85584b8b77-dcqq8" event={"ID":"463f10a4-6532-431c-b7a5-67472c923855","Type":"ContainerDied","Data":"ddcbebadc7be9ef3a0f1dc2abe141b6f2e97f7b75cacf674ed836d9b9585f5d1"} Nov 03 00:07:08.535800 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.535710 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85584b8b77-dcqq8" event={"ID":"463f10a4-6532-431c-b7a5-67472c923855","Type":"ContainerDied","Data":"cb5818c26f420d2057f48a46b5d61708de2185963cfe20c9f82215c108566f0a"} Nov 03 00:07:08.535800 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.535731 6400 scope.go:122] "RemoveContainer" containerID="ddcbebadc7be9ef3a0f1dc2abe141b6f2e97f7b75cacf674ed836d9b9585f5d1" Nov 03 00:07:08.544623 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.544609 6400 scope.go:122] "RemoveContainer" containerID="ddcbebadc7be9ef3a0f1dc2abe141b6f2e97f7b75cacf674ed836d9b9585f5d1" Nov 03 00:07:08.544920 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:07:08.544900 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddcbebadc7be9ef3a0f1dc2abe141b6f2e97f7b75cacf674ed836d9b9585f5d1\": container with ID starting with ddcbebadc7be9ef3a0f1dc2abe141b6f2e97f7b75cacf674ed836d9b9585f5d1 not found: ID does not exist" containerID="ddcbebadc7be9ef3a0f1dc2abe141b6f2e97f7b75cacf674ed836d9b9585f5d1" Nov 03 00:07:08.544997 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.544931 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddcbebadc7be9ef3a0f1dc2abe141b6f2e97f7b75cacf674ed836d9b9585f5d1"} err="failed to get container status \"ddcbebadc7be9ef3a0f1dc2abe141b6f2e97f7b75cacf674ed836d9b9585f5d1\": rpc error: code = NotFound desc = could not find container \"ddcbebadc7be9ef3a0f1dc2abe141b6f2e97f7b75cacf674ed836d9b9585f5d1\": container with ID starting with ddcbebadc7be9ef3a0f1dc2abe141b6f2e97f7b75cacf674ed836d9b9585f5d1 not found: ID does not exist" Nov 03 00:07:08.697811 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.697779 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85584b8b77-dcqq8"] Nov 03 00:07:08.747392 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:08.747364 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85584b8b77-dcqq8"] Nov 03 00:07:09.177360 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:09.177327 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="463f10a4-6532-431c-b7a5-67472c923855" path="/var/lib/kubelet/pods/463f10a4-6532-431c-b7a5-67472c923855/volumes" Nov 03 00:07:11.818361 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:11.818320 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:27.151521 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:07:27.151483 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:07:57.151129 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:07:57.151090 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:07:58.692690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:07:58.692651 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:05.898027 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:05.897988 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:06.194564 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:06.194450 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:06.862840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:06.862785 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q"] Nov 03 00:08:06.863167 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:06.863153 6400 cpu_manager.go:414] "containerMap: removing container" podUID="463f10a4-6532-431c-b7a5-67472c923855" containerName="console" Nov 03 00:08:06.863215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:06.863170 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463f10a4-6532-431c-b7a5-67472c923855" containerName="console" Nov 03 00:08:06.863257 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:06.863227 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="463f10a4-6532-431c-b7a5-67472c923855" containerName="console" Nov 03 00:08:06.865135 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:06.865118 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" Nov 03 00:08:06.917342 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:06.917318 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 00:08:06.959338 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:06.959305 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sph5r\" (UniqueName: \"kubernetes.io/projected/87c9c35c-0756-4944-ab43-c3c48f453c9a-kube-api-access-sph5r\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q\" (UID: \"87c9c35c-0756-4944-ab43-c3c48f453c9a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" Nov 03 00:08:06.959487 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:06.959378 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q\" (UID: \"87c9c35c-0756-4944-ab43-c3c48f453c9a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" Nov 03 00:08:06.959487 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:06.959476 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q\" (UID: \"87c9c35c-0756-4944-ab43-c3c48f453c9a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" Nov 03 00:08:06.966992 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:06.966964 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 00:08:06.967265 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:06.967247 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-9khbq\"" Nov 03 00:08:06.967904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:06.967887 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q"] Nov 03 00:08:07.060581 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.060551 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q\" (UID: \"87c9c35c-0756-4944-ab43-c3c48f453c9a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" Nov 03 00:08:07.060741 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.060603 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sph5r\" (UniqueName: \"kubernetes.io/projected/87c9c35c-0756-4944-ab43-c3c48f453c9a-kube-api-access-sph5r\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q\" (UID: \"87c9c35c-0756-4944-ab43-c3c48f453c9a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" Nov 03 00:08:07.060741 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.060663 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q\" (UID: \"87c9c35c-0756-4944-ab43-c3c48f453c9a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" Nov 03 00:08:07.061050 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.061031 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q\" (UID: \"87c9c35c-0756-4944-ab43-c3c48f453c9a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" Nov 03 00:08:07.061094 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.061050 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q\" (UID: \"87c9c35c-0756-4944-ab43-c3c48f453c9a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" Nov 03 00:08:07.075788 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.075759 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2"] Nov 03 00:08:07.077911 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.077897 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.118396 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.118340 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sph5r\" (UniqueName: \"kubernetes.io/projected/87c9c35c-0756-4944-ab43-c3c48f453c9a-kube-api-access-sph5r\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q\" (UID: \"87c9c35c-0756-4944-ab43-c3c48f453c9a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" Nov 03 00:08:07.133408 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.133389 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 00:08:07.134472 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.134459 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 00:08:07.134661 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.134646 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 00:08:07.140017 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.139997 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2"] Nov 03 00:08:07.161318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.161294 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.161423 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.161329 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sccw4\" (UniqueName: \"kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-kube-api-access-sccw4\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.161423 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.161359 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.161497 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.161448 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.161533 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.161492 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.161533 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.161521 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.161619 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.161551 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36b438b-02c6-4d4c-a201-05098c6f7f90-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.177415 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.177397 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" Nov 03 00:08:07.181978 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.181955 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:07.224220 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.224170 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gnssc\"" Nov 03 00:08:07.224220 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.224192 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 00:08:07.224220 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.224213 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 00:08:07.224471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.224214 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 00:08:07.262468 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.262439 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.262622 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.262474 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sccw4\" (UniqueName: \"kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-kube-api-access-sccw4\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.262622 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.262499 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.262710 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.262630 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.262710 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.262666 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.262710 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.262696 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.262885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.262731 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36b438b-02c6-4d4c-a201-05098c6f7f90-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.263745 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.263718 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.265117 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.265093 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.265217 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.265167 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36b438b-02c6-4d4c-a201-05098c6f7f90-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.265268 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.265229 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.265306 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.265276 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.318875 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.318850 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.319105 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.319085 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sccw4\" (UniqueName: \"kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-kube-api-access-sccw4\") pod \"istiod-openshift-gateway-7876bfcb74-stdh2\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.386457 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.386382 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:07.388187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.388169 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:07.546236 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.546203 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q"] Nov 03 00:08:07.549285 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:08:07.549255 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c9c35c_0756_4944_ab43_c3c48f453c9a.slice/crio-9a2f8e75ef6e9704ed64850de1852104ed50825925412db271208b7dd83d2d14 WatchSource:0}: Error finding container 9a2f8e75ef6e9704ed64850de1852104ed50825925412db271208b7dd83d2d14: Status 404 returned error can't find the container with id 9a2f8e75ef6e9704ed64850de1852104ed50825925412db271208b7dd83d2d14 Nov 03 00:08:07.749240 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.749156 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" event={"ID":"87c9c35c-0756-4944-ab43-c3c48f453c9a","Type":"ContainerStarted","Data":"9a2f8e75ef6e9704ed64850de1852104ed50825925412db271208b7dd83d2d14"} Nov 03 00:08:07.758528 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:07.758498 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2"] Nov 03 00:08:07.761401 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:08:07.761376 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36b438b_02c6_4d4c_a201_05098c6f7f90.slice/crio-80f6c4d9534df70e6cdfffa804e5063f9934537dbde90614432622ab2177d3fe WatchSource:0}: Error finding container 80f6c4d9534df70e6cdfffa804e5063f9934537dbde90614432622ab2177d3fe: Status 404 returned error can't find the container with id 80f6c4d9534df70e6cdfffa804e5063f9934537dbde90614432622ab2177d3fe Nov 03 00:08:08.132311 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:08.132281 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:08.757841 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:08.757776 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" event={"ID":"a36b438b-02c6-4d4c-a201-05098c6f7f90","Type":"ContainerStarted","Data":"80f6c4d9534df70e6cdfffa804e5063f9934537dbde90614432622ab2177d3fe"} Nov 03 00:08:14.785879 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:14.785836 6400 generic.go:356] "Generic (PLEG): container finished" podID="87c9c35c-0756-4944-ab43-c3c48f453c9a" containerID="412bd55a247baf318820073a384572cb84eb7100de6bcd016d7ddec8d57e6182" exitCode=0 Nov 03 00:08:14.786309 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:14.785921 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" event={"ID":"87c9c35c-0756-4944-ab43-c3c48f453c9a","Type":"ContainerDied","Data":"412bd55a247baf318820073a384572cb84eb7100de6bcd016d7ddec8d57e6182"} Nov 03 00:08:14.919498 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:14.919467 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:08:14.919589 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:14.919531 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:08:15.792147 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:15.792110 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" event={"ID":"a36b438b-02c6-4d4c-a201-05098c6f7f90","Type":"ContainerStarted","Data":"90394065e5ca899eda4f80b901c8600bcaeaa1184b1b9413b629eda3d5124dd9"} Nov 03 00:08:15.792565 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:15.792343 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:15.794249 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:15.794218 6400 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-stdh2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:08:15.794399 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:15.794273 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" podUID="a36b438b-02c6-4d4c-a201-05098c6f7f90" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:08:15.998848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:15.998773 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" podStartSLOduration=1.843374646 podStartE2EDuration="8.998757692s" totalImagesPullingTime="7.155383046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:15.958134377 +0000 UTC m=+142.909074536" watchObservedRunningTime="2026-11-03 00:08:15.998757692 +0000 UTC m=+142.949697849" Nov 03 00:08:16.243624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:16.243472 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:16.409536 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:16.409496 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:16.798015 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:16.797980 6400 generic.go:356] "Generic (PLEG): container finished" podID="87c9c35c-0756-4944-ab43-c3c48f453c9a" containerID="bb3ffece61345dfad2f372426252eabb73be5dcca8dc275008d21c28a528d5a9" exitCode=0 Nov 03 00:08:16.798408 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:16.798019 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" event={"ID":"87c9c35c-0756-4944-ab43-c3c48f453c9a","Type":"ContainerDied","Data":"bb3ffece61345dfad2f372426252eabb73be5dcca8dc275008d21c28a528d5a9"} Nov 03 00:08:16.798984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:16.798797 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:08:16.827008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:16.826983 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:17.144325 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:17.144286 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:17.367165 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:17.367080 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:20.804560 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:20.804518 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:24.840485 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:24.840455 6400 generic.go:356] "Generic (PLEG): container finished" podID="87c9c35c-0756-4944-ab43-c3c48f453c9a" containerID="823cc026044c749a0b6657971e43469f16e765afebfb3a7dc8e764ec0c9c9aef" exitCode=0 Nov 03 00:08:24.840909 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:24.840540 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" event={"ID":"87c9c35c-0756-4944-ab43-c3c48f453c9a","Type":"ContainerDied","Data":"823cc026044c749a0b6657971e43469f16e765afebfb3a7dc8e764ec0c9c9aef"} Nov 03 00:08:25.974638 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:25.974613 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" Nov 03 00:08:26.036941 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:26.036912 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c9c35c-0756-4944-ab43-c3c48f453c9a-kube-api-access-sph5r\" (UniqueName: \"kubernetes.io/projected/87c9c35c-0756-4944-ab43-c3c48f453c9a-kube-api-access-sph5r\") pod \"87c9c35c-0756-4944-ab43-c3c48f453c9a\" (UID: \"87c9c35c-0756-4944-ab43-c3c48f453c9a\") " Nov 03 00:08:26.037160 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:26.037009 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-bundle\") pod \"87c9c35c-0756-4944-ab43-c3c48f453c9a\" (UID: \"87c9c35c-0756-4944-ab43-c3c48f453c9a\") " Nov 03 00:08:26.037160 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:26.037108 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-util\" (UniqueName: \"kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-util\") pod \"87c9c35c-0756-4944-ab43-c3c48f453c9a\" (UID: \"87c9c35c-0756-4944-ab43-c3c48f453c9a\") " Nov 03 00:08:26.037572 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:26.037549 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-bundle" pod "87c9c35c-0756-4944-ab43-c3c48f453c9a" (UID: "87c9c35c-0756-4944-ab43-c3c48f453c9a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:26.039324 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:26.039303 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c9c35c-0756-4944-ab43-c3c48f453c9a-kube-api-access-sph5r" pod "87c9c35c-0756-4944-ab43-c3c48f453c9a" (UID: "87c9c35c-0756-4944-ab43-c3c48f453c9a"). InnerVolumeSpecName "kube-api-access-sph5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:26.043633 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:26.043605 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-util" pod "87c9c35c-0756-4944-ab43-c3c48f453c9a" (UID: "87c9c35c-0756-4944-ab43-c3c48f453c9a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:26.138096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:26.138004 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sph5r\" (UniqueName: \"kubernetes.io/projected/87c9c35c-0756-4944-ab43-c3c48f453c9a-kube-api-access-sph5r\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:26.138096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:26.138039 6400 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-bundle\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:26.138096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:26.138048 6400 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87c9c35c-0756-4944-ab43-c3c48f453c9a-util\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:26.849991 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:26.849954 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" event={"ID":"87c9c35c-0756-4944-ab43-c3c48f453c9a","Type":"ContainerDied","Data":"9a2f8e75ef6e9704ed64850de1852104ed50825925412db271208b7dd83d2d14"} Nov 03 00:08:26.849991 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:26.849972 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mfp5q" Nov 03 00:08:26.850218 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:26.850005 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a2f8e75ef6e9704ed64850de1852104ed50825925412db271208b7dd83d2d14" Nov 03 00:08:27.153563 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:08:27.153467 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:08:39.544268 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:39.544231 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:40.582577 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.582541 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz"] Nov 03 00:08:40.582997 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.582873 6400 cpu_manager.go:414] "containerMap: removing container" podUID="87c9c35c-0756-4944-ab43-c3c48f453c9a" containerName="util" Nov 03 00:08:40.582997 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.582884 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c9c35c-0756-4944-ab43-c3c48f453c9a" containerName="util" Nov 03 00:08:40.582997 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.582899 6400 cpu_manager.go:414] "containerMap: removing container" podUID="87c9c35c-0756-4944-ab43-c3c48f453c9a" containerName="pull" Nov 03 00:08:40.582997 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.582904 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c9c35c-0756-4944-ab43-c3c48f453c9a" containerName="pull" Nov 03 00:08:40.582997 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.582926 6400 cpu_manager.go:414] "containerMap: removing container" podUID="87c9c35c-0756-4944-ab43-c3c48f453c9a" containerName="extract" Nov 03 00:08:40.582997 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.582932 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c9c35c-0756-4944-ab43-c3c48f453c9a" containerName="extract" Nov 03 00:08:40.582997 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.582995 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c9c35c-0756-4944-ab43-c3c48f453c9a" containerName="extract" Nov 03 00:08:40.584942 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.584926 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.662652 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.662622 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c72f732d-c3c4-4c77-b129-5cc9bc75c570-metrics-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.662812 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.662669 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c72f732d-c3c4-4c77-b129-5cc9bc75c570-tmp\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.662812 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.662728 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c72f732d-c3c4-4c77-b129-5cc9bc75c570-manifests\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.662812 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.662744 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qm2\" (UniqueName: \"kubernetes.io/projected/c72f732d-c3c4-4c77-b129-5cc9bc75c570-kube-api-access-69qm2\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.662812 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.662778 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c72f732d-c3c4-4c77-b129-5cc9bc75c570-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.663021 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.662861 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c72f732d-c3c4-4c77-b129-5cc9bc75c570-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.687086 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.687044 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 00:08:40.687086 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.687048 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 00:08:40.687265 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.687097 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9xr6v\"" Nov 03 00:08:40.687265 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.687060 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 00:08:40.687265 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.687050 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 00:08:40.687265 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.687142 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 00:08:40.688122 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.688105 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz"] Nov 03 00:08:40.764235 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.764208 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c72f732d-c3c4-4c77-b129-5cc9bc75c570-metrics-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.764396 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.764248 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c72f732d-c3c4-4c77-b129-5cc9bc75c570-tmp\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.764396 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.764288 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c72f732d-c3c4-4c77-b129-5cc9bc75c570-manifests\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.764396 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.764312 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qm2\" (UniqueName: \"kubernetes.io/projected/c72f732d-c3c4-4c77-b129-5cc9bc75c570-kube-api-access-69qm2\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.764396 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.764368 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c72f732d-c3c4-4c77-b129-5cc9bc75c570-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.764624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.764399 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c72f732d-c3c4-4c77-b129-5cc9bc75c570-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.764684 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.764666 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c72f732d-c3c4-4c77-b129-5cc9bc75c570-manifests\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.766915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.766883 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c72f732d-c3c4-4c77-b129-5cc9bc75c570-tmp\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.767035 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.766992 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c72f732d-c3c4-4c77-b129-5cc9bc75c570-metrics-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.767095 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.767084 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c72f732d-c3c4-4c77-b129-5cc9bc75c570-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.767150 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.767098 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c72f732d-c3c4-4c77-b129-5cc9bc75c570-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.821499 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.821465 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qm2\" (UniqueName: \"kubernetes.io/projected/c72f732d-c3c4-4c77-b129-5cc9bc75c570-kube-api-access-69qm2\") pod \"opendatahub-operator-controller-manager-65f97f758d-pj5sz\" (UID: \"c72f732d-c3c4-4c77-b129-5cc9bc75c570\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:40.897712 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:40.897614 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:41.304800 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:41.304769 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:41.384259 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:41.384233 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz"] Nov 03 00:08:41.387365 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:08:41.387336 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72f732d_c3c4_4c77_b129_5cc9bc75c570.slice/crio-4eaf833a6489c06f971fb6ec3bc6f7b16cc0736556d324606336712f892dc888 WatchSource:0}: Error finding container 4eaf833a6489c06f971fb6ec3bc6f7b16cc0736556d324606336712f892dc888: Status 404 returned error can't find the container with id 4eaf833a6489c06f971fb6ec3bc6f7b16cc0736556d324606336712f892dc888 Nov 03 00:08:41.906784 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:41.906744 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" event={"ID":"c72f732d-c3c4-4c77-b129-5cc9bc75c570","Type":"ContainerStarted","Data":"4eaf833a6489c06f971fb6ec3bc6f7b16cc0736556d324606336712f892dc888"} Nov 03 00:08:44.132933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:44.132894 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:44.352153 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:44.352108 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:45.927619 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:45.927585 6400 generic.go:356] "Generic (PLEG): container finished" podID="c72f732d-c3c4-4c77-b129-5cc9bc75c570" containerID="c654ffac579ff2475292f33cfd44b9e6cfc08dcf3eac46d9eccd5054d2c3ed3d" exitCode=0 Nov 03 00:08:45.928064 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:45.927667 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" event={"ID":"c72f732d-c3c4-4c77-b129-5cc9bc75c570","Type":"ContainerDied","Data":"c654ffac579ff2475292f33cfd44b9e6cfc08dcf3eac46d9eccd5054d2c3ed3d"} Nov 03 00:08:46.933750 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:46.933704 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" event={"ID":"c72f732d-c3c4-4c77-b129-5cc9bc75c570","Type":"ContainerStarted","Data":"9b1ce3603bab4fa1a1d7ae3316cbb059eb1d8028e65a91145efd850ce108f425"} Nov 03 00:08:46.934156 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:46.933875 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:47.168142 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:47.168028 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" podStartSLOduration=2.062671119 podStartE2EDuration="7.168014447s" totalImagesPullingTime="4.646996968s" totalInitContainerRuntime="458.34636ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:47.118638451 +0000 UTC m=+174.069578645" watchObservedRunningTime="2026-11-03 00:08:47.168014447 +0000 UTC m=+174.118954662" Nov 03 00:08:48.516334 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:48.516287 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:49.189538 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:49.189505 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:50.216926 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:50.216859 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:50.410586 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:50.410540 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:52.830929 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:52.830895 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:53.705634 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:53.705594 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:54.114530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:54.114501 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:54.497069 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:54.496983 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:55.181403 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:55.181366 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:55.462964 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:55.462881 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:57.151694 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:08:57.151648 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:08:57.940240 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:57.940209 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-pj5sz" Nov 03 00:08:58.223302 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:58.223219 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:58.545015 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:58.544972 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:58.711756 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:58.711715 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:59.063176 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:59.063140 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:59.437125 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:59.437042 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:59.657757 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:08:59.657719 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:02.312650 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:02.312564 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:07.165207 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:07.165165 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:07.348111 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:07.348064 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:08.320412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:08.320366 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:08.478490 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:08.478462 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:08.994866 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:08.994807 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:10.042038 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:10.041999 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:11.842511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:11.842475 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:14.100479 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:14.100427 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:14.308732 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:14.308690 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:15.305914 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:15.305881 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:15.536250 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:15.536214 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:17.605366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:17.605334 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:17.896507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:17.896405 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:18.111380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.111346 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:18.339875 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.339836 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:18.714919 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.714807 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:18.839570 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.839524 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh"] Nov 03 00:09:18.846694 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.846671 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh" Nov 03 00:09:18.881027 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.880998 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:18.885001 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.884968 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-zzvjh\" (UID: \"fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh" Nov 03 00:09:18.885082 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.885019 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r626h\" (UniqueName: \"kubernetes.io/projected/fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4-kube-api-access-r626h\") pod \"cert-manager-webhook-5dbbbf75cc-zzvjh\" (UID: \"fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh" Nov 03 00:09:18.948150 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.948122 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-g524k\"" Nov 03 00:09:18.948150 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.948132 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 00:09:18.948348 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.948335 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 00:09:18.949465 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.949450 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh"] Nov 03 00:09:18.985868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.985799 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-zzvjh\" (UID: \"fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh" Nov 03 00:09:18.985981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:18.985868 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r626h\" (UniqueName: \"kubernetes.io/projected/fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4-kube-api-access-r626h\") pod \"cert-manager-webhook-5dbbbf75cc-zzvjh\" (UID: \"fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh" Nov 03 00:09:19.044116 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:19.044088 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-zzvjh\" (UID: \"fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh" Nov 03 00:09:19.044277 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:19.044257 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r626h\" (UniqueName: \"kubernetes.io/projected/fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4-kube-api-access-r626h\") pod \"cert-manager-webhook-5dbbbf75cc-zzvjh\" (UID: \"fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh" Nov 03 00:09:19.167804 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:19.167780 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh" Nov 03 00:09:19.524264 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:19.524235 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:19.541237 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:19.541216 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh"] Nov 03 00:09:19.544515 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:09:19.544483 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc2f569e_2f6f_45e1_8cc7_9a73090fa3d4.slice/crio-bcdd54120789a510b8d32d2b4d908d5c054f39a6bc6367d2570b8f492d4d20fd WatchSource:0}: Error finding container bcdd54120789a510b8d32d2b4d908d5c054f39a6bc6367d2570b8f492d4d20fd: Status 404 returned error can't find the container with id bcdd54120789a510b8d32d2b4d908d5c054f39a6bc6367d2570b8f492d4d20fd Nov 03 00:09:19.681658 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:19.681627 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:19.849235 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:19.849192 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:20.004459 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:20.004424 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:20.074699 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:20.074664 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh" event={"ID":"fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4","Type":"ContainerStarted","Data":"bcdd54120789a510b8d32d2b4d908d5c054f39a6bc6367d2570b8f492d4d20fd"} Nov 03 00:09:20.560190 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:20.560158 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:20.715187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:20.715148 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:21.127487 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.127449 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:21.179676 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.179648 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dnw7d"] Nov 03 00:09:21.182084 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.182050 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" Nov 03 00:09:21.205550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.205518 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4ace7ca4-13ad-42a0-9881-edc2712e7502-config-file\") pod \"limitador-limitador-74dfcfb894-dnw7d\" (UID: \"4ace7ca4-13ad-42a0-9881-edc2712e7502\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" Nov 03 00:09:21.205700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.205655 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rszq2\" (UniqueName: \"kubernetes.io/projected/4ace7ca4-13ad-42a0-9881-edc2712e7502-kube-api-access-rszq2\") pod \"limitador-limitador-74dfcfb894-dnw7d\" (UID: \"4ace7ca4-13ad-42a0-9881-edc2712e7502\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" Nov 03 00:09:21.232358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.232331 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dnw7d"] Nov 03 00:09:21.279977 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.279947 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 00:09:21.279977 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.279970 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 00:09:21.280186 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.279969 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 00:09:21.286877 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.286853 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9fpwb\"" Nov 03 00:09:21.306847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.306793 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4ace7ca4-13ad-42a0-9881-edc2712e7502-config-file\") pod \"limitador-limitador-74dfcfb894-dnw7d\" (UID: \"4ace7ca4-13ad-42a0-9881-edc2712e7502\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" Nov 03 00:09:21.307076 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.306901 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rszq2\" (UniqueName: \"kubernetes.io/projected/4ace7ca4-13ad-42a0-9881-edc2712e7502-kube-api-access-rszq2\") pod \"limitador-limitador-74dfcfb894-dnw7d\" (UID: \"4ace7ca4-13ad-42a0-9881-edc2712e7502\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" Nov 03 00:09:21.307544 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.307521 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4ace7ca4-13ad-42a0-9881-edc2712e7502-config-file\") pod \"limitador-limitador-74dfcfb894-dnw7d\" (UID: \"4ace7ca4-13ad-42a0-9881-edc2712e7502\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" Nov 03 00:09:21.337601 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.337567 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:21.364182 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.364099 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rszq2\" (UniqueName: \"kubernetes.io/projected/4ace7ca4-13ad-42a0-9881-edc2712e7502-kube-api-access-rszq2\") pod \"limitador-limitador-74dfcfb894-dnw7d\" (UID: \"4ace7ca4-13ad-42a0-9881-edc2712e7502\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" Nov 03 00:09:21.414752 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.414673 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dnw7d"] Nov 03 00:09:21.497810 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.497780 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" Nov 03 00:09:21.498523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.498495 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:21.882137 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:21.882104 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dnw7d"] Nov 03 00:09:22.087962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:22.087921 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" event={"ID":"4ace7ca4-13ad-42a0-9881-edc2712e7502","Type":"ContainerStarted","Data":"6ae2f4adfcc2dbf5af92ea025f02125147aec01762d47e0fa163615dbe2dce9d"} Nov 03 00:09:22.448563 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:22.448525 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wfs24"] Nov 03 00:09:22.456225 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:22.456196 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wfs24" Nov 03 00:09:22.506657 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:22.506627 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wfs24"] Nov 03 00:09:22.519017 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:22.518986 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfs5\" (UniqueName: \"kubernetes.io/projected/30de02c8-b06e-4a94-8185-f30880b602e0-kube-api-access-rvfs5\") pod \"authorino-57f7f8659f-wfs24\" (UID: \"30de02c8-b06e-4a94-8185-f30880b602e0\") " pod="kuadrant-system/authorino-57f7f8659f-wfs24" Nov 03 00:09:22.572594 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:22.572559 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7ns8x\"" Nov 03 00:09:22.619633 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:22.619594 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfs5\" (UniqueName: \"kubernetes.io/projected/30de02c8-b06e-4a94-8185-f30880b602e0-kube-api-access-rvfs5\") pod \"authorino-57f7f8659f-wfs24\" (UID: \"30de02c8-b06e-4a94-8185-f30880b602e0\") " pod="kuadrant-system/authorino-57f7f8659f-wfs24" Nov 03 00:09:22.684396 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:22.684369 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfs5\" (UniqueName: \"kubernetes.io/projected/30de02c8-b06e-4a94-8185-f30880b602e0-kube-api-access-rvfs5\") pod \"authorino-57f7f8659f-wfs24\" (UID: \"30de02c8-b06e-4a94-8185-f30880b602e0\") " pod="kuadrant-system/authorino-57f7f8659f-wfs24" Nov 03 00:09:22.687058 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:22.687025 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:22.769794 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:22.769702 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wfs24" Nov 03 00:09:22.897074 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:22.897031 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:23.166430 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:23.166395 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wfs24"] Nov 03 00:09:23.169916 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:09:23.169888 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30de02c8_b06e_4a94_8185_f30880b602e0.slice/crio-620699d73df246d3d609a9536044c9e2b16cea8411dea8c5630b6fa12bc06b26 WatchSource:0}: Error finding container 620699d73df246d3d609a9536044c9e2b16cea8411dea8c5630b6fa12bc06b26: Status 404 returned error can't find the container with id 620699d73df246d3d609a9536044c9e2b16cea8411dea8c5630b6fa12bc06b26 Nov 03 00:09:24.100681 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:24.100643 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wfs24" event={"ID":"30de02c8-b06e-4a94-8185-f30880b602e0","Type":"ContainerStarted","Data":"620699d73df246d3d609a9536044c9e2b16cea8411dea8c5630b6fa12bc06b26"} Nov 03 00:09:24.678193 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:24.678158 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:24.877698 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:24.877643 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:25.109888 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.109053 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh" event={"ID":"fc2f569e-2f6f-45e1-8cc7-9a73090fa3d4","Type":"ContainerStarted","Data":"4a0c0fbb8d43034181e2fd944f3860d09263a1735ab7fbc92f121fcf40be3004"} Nov 03 00:09:25.109888 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.109837 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh" Nov 03 00:09:25.234184 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.234123 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh" podStartSLOduration=2.7267018910000003 podStartE2EDuration="7.23410122s" totalImagesPullingTime="4.507399329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:25.23380232 +0000 UTC m=+212.184742480" watchObservedRunningTime="2026-11-03 00:09:25.23410122 +0000 UTC m=+212.185041378" Nov 03 00:09:25.570395 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.570357 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-p5fb4"] Nov 03 00:09:25.574197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.574160 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-p5fb4" Nov 03 00:09:25.581173 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.581149 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:09:25.584427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.584406 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-8d9rl" Nov 03 00:09:25.630704 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.630668 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-p5fb4"] Nov 03 00:09:25.652775 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.652740 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v57k\" (UniqueName: \"kubernetes.io/projected/9cb77863-cd9c-418b-91e3-96ceec1fd8fc-kube-api-access-2v57k\") pod \"authorino-56c6c9ddd5-p5fb4\" (UID: \"9cb77863-cd9c-418b-91e3-96ceec1fd8fc\") " pod="kuadrant-system/authorino-56c6c9ddd5-p5fb4" Nov 03 00:09:25.652970 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.652788 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc80ce75-cd2c-43c0-888b-4e41bbd9e047-config-file\") pod \"limitador-limitador-7c9b87d574-8d9rl\" (UID: \"fc80ce75-cd2c-43c0-888b-4e41bbd9e047\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8d9rl" Nov 03 00:09:25.652970 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.652865 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9f7r\" (UniqueName: \"kubernetes.io/projected/fc80ce75-cd2c-43c0-888b-4e41bbd9e047-kube-api-access-d9f7r\") pod \"limitador-limitador-7c9b87d574-8d9rl\" (UID: \"fc80ce75-cd2c-43c0-888b-4e41bbd9e047\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8d9rl" Nov 03 00:09:25.672624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.669903 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:09:25.754057 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.754030 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v57k\" (UniqueName: \"kubernetes.io/projected/9cb77863-cd9c-418b-91e3-96ceec1fd8fc-kube-api-access-2v57k\") pod \"authorino-56c6c9ddd5-p5fb4\" (UID: \"9cb77863-cd9c-418b-91e3-96ceec1fd8fc\") " pod="kuadrant-system/authorino-56c6c9ddd5-p5fb4" Nov 03 00:09:25.754427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.754073 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc80ce75-cd2c-43c0-888b-4e41bbd9e047-config-file\") pod \"limitador-limitador-7c9b87d574-8d9rl\" (UID: \"fc80ce75-cd2c-43c0-888b-4e41bbd9e047\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8d9rl" Nov 03 00:09:25.754427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.754117 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9f7r\" (UniqueName: \"kubernetes.io/projected/fc80ce75-cd2c-43c0-888b-4e41bbd9e047-kube-api-access-d9f7r\") pod \"limitador-limitador-7c9b87d574-8d9rl\" (UID: \"fc80ce75-cd2c-43c0-888b-4e41bbd9e047\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8d9rl" Nov 03 00:09:25.754949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.754928 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc80ce75-cd2c-43c0-888b-4e41bbd9e047-config-file\") pod \"limitador-limitador-7c9b87d574-8d9rl\" (UID: \"fc80ce75-cd2c-43c0-888b-4e41bbd9e047\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8d9rl" Nov 03 00:09:25.811436 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.811414 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v57k\" (UniqueName: \"kubernetes.io/projected/9cb77863-cd9c-418b-91e3-96ceec1fd8fc-kube-api-access-2v57k\") pod \"authorino-56c6c9ddd5-p5fb4\" (UID: \"9cb77863-cd9c-418b-91e3-96ceec1fd8fc\") " pod="kuadrant-system/authorino-56c6c9ddd5-p5fb4" Nov 03 00:09:25.812489 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.812466 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9f7r\" (UniqueName: \"kubernetes.io/projected/fc80ce75-cd2c-43c0-888b-4e41bbd9e047-kube-api-access-d9f7r\") pod \"limitador-limitador-7c9b87d574-8d9rl\" (UID: \"fc80ce75-cd2c-43c0-888b-4e41bbd9e047\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8d9rl" Nov 03 00:09:25.872704 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.872571 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:09:25.889144 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.889113 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-p5fb4" Nov 03 00:09:25.899503 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:25.899482 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-8d9rl" Nov 03 00:09:26.114318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:26.114282 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" event={"ID":"4ace7ca4-13ad-42a0-9881-edc2712e7502","Type":"ContainerStarted","Data":"83ae38fa9e2950f22f1cc6ff54d6c467e104b8f89bbc5a132629d25b14a745b0"} Nov 03 00:09:26.114710 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:26.114405 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" Nov 03 00:09:26.227354 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:26.227258 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" podStartSLOduration=1.368181093 podStartE2EDuration="5.227242444s" totalImagesPullingTime="3.859061351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:26.225839033 +0000 UTC m=+213.176779182" watchObservedRunningTime="2026-11-03 00:09:26.227242444 +0000 UTC m=+213.178182602" Nov 03 00:09:26.276094 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:26.276062 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-p5fb4"] Nov 03 00:09:26.279182 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:09:26.279156 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cb77863_cd9c_418b_91e3_96ceec1fd8fc.slice/crio-b906b13783da032d9cceec7a97d359a7d36c1195ce5f2a41d428df8bdbf499bd WatchSource:0}: Error finding container b906b13783da032d9cceec7a97d359a7d36c1195ce5f2a41d428df8bdbf499bd: Status 404 returned error can't find the container with id b906b13783da032d9cceec7a97d359a7d36c1195ce5f2a41d428df8bdbf499bd Nov 03 00:09:26.326967 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:26.326943 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:09:26.329911 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:09:26.329882 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc80ce75_cd2c_43c0_888b_4e41bbd9e047.slice/crio-9cf88d1a719eb07048c2a0b76a203e0804eb3cf837b0cc9da1f405019ccdcb42 WatchSource:0}: Error finding container 9cf88d1a719eb07048c2a0b76a203e0804eb3cf837b0cc9da1f405019ccdcb42: Status 404 returned error can't find the container with id 9cf88d1a719eb07048c2a0b76a203e0804eb3cf837b0cc9da1f405019ccdcb42 Nov 03 00:09:26.536714 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:26.536685 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:26.702123 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:26.702080 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:27.120810 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:27.120764 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-p5fb4" event={"ID":"9cb77863-cd9c-418b-91e3-96ceec1fd8fc","Type":"ContainerStarted","Data":"b906b13783da032d9cceec7a97d359a7d36c1195ce5f2a41d428df8bdbf499bd"} Nov 03 00:09:27.124922 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:27.124884 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-8d9rl" event={"ID":"fc80ce75-cd2c-43c0-888b-4e41bbd9e047","Type":"ContainerStarted","Data":"75ba9197be19ffbf6fd1073a3355d2e4fe49f6c46d2f90abd9729848a8fe8b36"} Nov 03 00:09:27.125064 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:27.124969 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-8d9rl" event={"ID":"fc80ce75-cd2c-43c0-888b-4e41bbd9e047","Type":"ContainerStarted","Data":"9cf88d1a719eb07048c2a0b76a203e0804eb3cf837b0cc9da1f405019ccdcb42"} Nov 03 00:09:27.125653 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:27.125632 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-8d9rl" Nov 03 00:09:27.161247 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:09:27.161209 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:09:27.238132 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:27.238074 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-8d9rl" podStartSLOduration=2.238055666 podStartE2EDuration="2.238055666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:27.236721467 +0000 UTC m=+214.187661626" watchObservedRunningTime="2026-11-03 00:09:27.238055666 +0000 UTC m=+214.188995826" Nov 03 00:09:27.999638 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:27.999605 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:28.129905 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:28.129872 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-p5fb4" event={"ID":"9cb77863-cd9c-418b-91e3-96ceec1fd8fc","Type":"ContainerStarted","Data":"59eb60cf6b64aa5d655d1536c10f6a5a5769e8ee3335c735cdd95e8ef4196434"} Nov 03 00:09:28.131324 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:28.131300 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wfs24" event={"ID":"30de02c8-b06e-4a94-8185-f30880b602e0","Type":"ContainerStarted","Data":"031dfe5c76c8eba26a2e29177bd89f38da9a76bad6f13055b3fd78c8d9cde497"} Nov 03 00:09:28.246379 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:28.246322 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-p5fb4" podStartSLOduration=2.082765018 podStartE2EDuration="3.24630444s" totalImagesPullingTime="1.163539422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:28.245703234 +0000 UTC m=+215.196643393" watchObservedRunningTime="2026-11-03 00:09:28.24630444 +0000 UTC m=+215.197244592" Nov 03 00:09:28.271667 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:28.271583 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wfs24"] Nov 03 00:09:28.357172 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:28.357114 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-wfs24" podStartSLOduration=2.085170972 podStartE2EDuration="6.357101293s" totalImagesPullingTime="4.271930321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:28.3564574 +0000 UTC m=+215.307397556" watchObservedRunningTime="2026-11-03 00:09:28.357101293 +0000 UTC m=+215.308041451" Nov 03 00:09:29.038311 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:29.038276 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:29.207029 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:29.206993 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:30.141266 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:30.141224 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-wfs24" podUID="30de02c8-b06e-4a94-8185-f30880b602e0" containerName="authorino" containerID="cri-o://031dfe5c76c8eba26a2e29177bd89f38da9a76bad6f13055b3fd78c8d9cde497" gracePeriod=30 Nov 03 00:09:30.383878 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:30.383855 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wfs24" Nov 03 00:09:30.501253 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:30.501123 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30de02c8-b06e-4a94-8185-f30880b602e0-kube-api-access-rvfs5\" (UniqueName: \"kubernetes.io/projected/30de02c8-b06e-4a94-8185-f30880b602e0-kube-api-access-rvfs5\") pod \"30de02c8-b06e-4a94-8185-f30880b602e0\" (UID: \"30de02c8-b06e-4a94-8185-f30880b602e0\") " Nov 03 00:09:30.503538 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:30.503510 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30de02c8-b06e-4a94-8185-f30880b602e0-kube-api-access-rvfs5" pod "30de02c8-b06e-4a94-8185-f30880b602e0" (UID: "30de02c8-b06e-4a94-8185-f30880b602e0"). InnerVolumeSpecName "kube-api-access-rvfs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:30.602216 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:30.602180 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvfs5\" (UniqueName: \"kubernetes.io/projected/30de02c8-b06e-4a94-8185-f30880b602e0-kube-api-access-rvfs5\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:31.147509 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:31.147474 6400 generic.go:356] "Generic (PLEG): container finished" podID="30de02c8-b06e-4a94-8185-f30880b602e0" containerID="031dfe5c76c8eba26a2e29177bd89f38da9a76bad6f13055b3fd78c8d9cde497" exitCode=0 Nov 03 00:09:31.147656 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:31.147520 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wfs24" Nov 03 00:09:31.147656 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:31.147559 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wfs24" event={"ID":"30de02c8-b06e-4a94-8185-f30880b602e0","Type":"ContainerDied","Data":"031dfe5c76c8eba26a2e29177bd89f38da9a76bad6f13055b3fd78c8d9cde497"} Nov 03 00:09:31.147656 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:31.147597 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wfs24" event={"ID":"30de02c8-b06e-4a94-8185-f30880b602e0","Type":"ContainerDied","Data":"620699d73df246d3d609a9536044c9e2b16cea8411dea8c5630b6fa12bc06b26"} Nov 03 00:09:31.147656 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:31.147615 6400 scope.go:122] "RemoveContainer" containerID="031dfe5c76c8eba26a2e29177bd89f38da9a76bad6f13055b3fd78c8d9cde497" Nov 03 00:09:31.156503 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:31.156487 6400 scope.go:122] "RemoveContainer" containerID="031dfe5c76c8eba26a2e29177bd89f38da9a76bad6f13055b3fd78c8d9cde497" Nov 03 00:09:31.156770 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:09:31.156751 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031dfe5c76c8eba26a2e29177bd89f38da9a76bad6f13055b3fd78c8d9cde497\": container with ID starting with 031dfe5c76c8eba26a2e29177bd89f38da9a76bad6f13055b3fd78c8d9cde497 not found: ID does not exist" containerID="031dfe5c76c8eba26a2e29177bd89f38da9a76bad6f13055b3fd78c8d9cde497" Nov 03 00:09:31.156893 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:31.156778 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031dfe5c76c8eba26a2e29177bd89f38da9a76bad6f13055b3fd78c8d9cde497"} err="failed to get container status \"031dfe5c76c8eba26a2e29177bd89f38da9a76bad6f13055b3fd78c8d9cde497\": rpc error: code = NotFound desc = could not find container \"031dfe5c76c8eba26a2e29177bd89f38da9a76bad6f13055b3fd78c8d9cde497\": container with ID starting with 031dfe5c76c8eba26a2e29177bd89f38da9a76bad6f13055b3fd78c8d9cde497 not found: ID does not exist" Nov 03 00:09:31.308210 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:31.308182 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wfs24"] Nov 03 00:09:31.311187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:31.311163 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wfs24"] Nov 03 00:09:32.126531 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:32.126502 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zzvjh" Nov 03 00:09:33.182472 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:33.182439 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30de02c8-b06e-4a94-8185-f30880b602e0" path="/var/lib/kubelet/pods/30de02c8-b06e-4a94-8185-f30880b602e0/volumes" Nov 03 00:09:33.654359 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:33.654328 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:33.857206 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:33.857167 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:37.126579 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:37.126551 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" Nov 03 00:09:38.384986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:38.384946 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:38.589409 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:38.589374 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:39.137559 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:39.137525 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-8d9rl" Nov 03 00:09:39.280900 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:39.280871 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dnw7d"] Nov 03 00:09:39.281093 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:39.281071 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" podUID="4ace7ca4-13ad-42a0-9881-edc2712e7502" containerName="limitador" containerID="cri-o://83ae38fa9e2950f22f1cc6ff54d6c467e104b8f89bbc5a132629d25b14a745b0" gracePeriod=30 Nov 03 00:09:39.831028 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:39.831006 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" Nov 03 00:09:39.886951 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:39.886884 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ace7ca4-13ad-42a0-9881-edc2712e7502-kube-api-access-rszq2\" (UniqueName: \"kubernetes.io/projected/4ace7ca4-13ad-42a0-9881-edc2712e7502-kube-api-access-rszq2\") pod \"4ace7ca4-13ad-42a0-9881-edc2712e7502\" (UID: \"4ace7ca4-13ad-42a0-9881-edc2712e7502\") " Nov 03 00:09:39.887071 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:39.886955 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ace7ca4-13ad-42a0-9881-edc2712e7502-config-file\" (UniqueName: \"kubernetes.io/configmap/4ace7ca4-13ad-42a0-9881-edc2712e7502-config-file\") pod \"4ace7ca4-13ad-42a0-9881-edc2712e7502\" (UID: \"4ace7ca4-13ad-42a0-9881-edc2712e7502\") " Nov 03 00:09:39.887332 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:39.887303 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ace7ca4-13ad-42a0-9881-edc2712e7502-config-file" pod "4ace7ca4-13ad-42a0-9881-edc2712e7502" (UID: "4ace7ca4-13ad-42a0-9881-edc2712e7502"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:09:39.889128 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:39.889110 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ace7ca4-13ad-42a0-9881-edc2712e7502-kube-api-access-rszq2" pod "4ace7ca4-13ad-42a0-9881-edc2712e7502" (UID: "4ace7ca4-13ad-42a0-9881-edc2712e7502"). InnerVolumeSpecName "kube-api-access-rszq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:39.988325 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:39.988290 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rszq2\" (UniqueName: \"kubernetes.io/projected/4ace7ca4-13ad-42a0-9881-edc2712e7502-kube-api-access-rszq2\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:39.988325 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:39.988319 6400 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4ace7ca4-13ad-42a0-9881-edc2712e7502-config-file\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:40.189020 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:40.188931 6400 generic.go:356] "Generic (PLEG): container finished" podID="4ace7ca4-13ad-42a0-9881-edc2712e7502" containerID="83ae38fa9e2950f22f1cc6ff54d6c467e104b8f89bbc5a132629d25b14a745b0" exitCode=0 Nov 03 00:09:40.189020 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:40.188978 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" Nov 03 00:09:40.189020 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:40.189009 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" event={"ID":"4ace7ca4-13ad-42a0-9881-edc2712e7502","Type":"ContainerDied","Data":"83ae38fa9e2950f22f1cc6ff54d6c467e104b8f89bbc5a132629d25b14a745b0"} Nov 03 00:09:40.189267 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:40.189053 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-dnw7d" event={"ID":"4ace7ca4-13ad-42a0-9881-edc2712e7502","Type":"ContainerDied","Data":"6ae2f4adfcc2dbf5af92ea025f02125147aec01762d47e0fa163615dbe2dce9d"} Nov 03 00:09:40.189267 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:40.189077 6400 scope.go:122] "RemoveContainer" containerID="83ae38fa9e2950f22f1cc6ff54d6c467e104b8f89bbc5a132629d25b14a745b0" Nov 03 00:09:40.198084 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:40.198066 6400 scope.go:122] "RemoveContainer" containerID="83ae38fa9e2950f22f1cc6ff54d6c467e104b8f89bbc5a132629d25b14a745b0" Nov 03 00:09:40.198348 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:09:40.198331 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ae38fa9e2950f22f1cc6ff54d6c467e104b8f89bbc5a132629d25b14a745b0\": container with ID starting with 83ae38fa9e2950f22f1cc6ff54d6c467e104b8f89bbc5a132629d25b14a745b0 not found: ID does not exist" containerID="83ae38fa9e2950f22f1cc6ff54d6c467e104b8f89bbc5a132629d25b14a745b0" Nov 03 00:09:40.198393 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:40.198357 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ae38fa9e2950f22f1cc6ff54d6c467e104b8f89bbc5a132629d25b14a745b0"} err="failed to get container status \"83ae38fa9e2950f22f1cc6ff54d6c467e104b8f89bbc5a132629d25b14a745b0\": rpc error: code = NotFound desc = could not find container \"83ae38fa9e2950f22f1cc6ff54d6c467e104b8f89bbc5a132629d25b14a745b0\": container with ID starting with 83ae38fa9e2950f22f1cc6ff54d6c467e104b8f89bbc5a132629d25b14a745b0 not found: ID does not exist" Nov 03 00:09:40.350146 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:40.350116 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dnw7d"] Nov 03 00:09:40.353604 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:40.353581 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dnw7d"] Nov 03 00:09:41.177358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:41.177315 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ace7ca4-13ad-42a0-9881-edc2712e7502" path="/var/lib/kubelet/pods/4ace7ca4-13ad-42a0-9881-edc2712e7502/volumes" Nov 03 00:09:45.664945 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:45.664906 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:45.854667 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:45.854628 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:46.412413 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:46.412375 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:48.183418 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:48.183387 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:48.388696 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:48.388655 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:52.624303 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:52.624255 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:52.807306 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:52.807269 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:57.152603 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:09:57.152562 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:09:57.626316 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:57.626286 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:57.842250 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:57.842213 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:59.392520 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:09:59.392483 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:02.829331 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:02.829285 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:03.070891 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:03.070857 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:03.943267 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:03.943231 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:04.138098 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:04.138059 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:07.564131 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.564098 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:07.590194 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.590156 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-857dc8657b-4wsbd"] Nov 03 00:10:07.590602 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.590584 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4ace7ca4-13ad-42a0-9881-edc2712e7502" containerName="limitador" Nov 03 00:10:07.590695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.590604 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ace7ca4-13ad-42a0-9881-edc2712e7502" containerName="limitador" Nov 03 00:10:07.590695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.590623 6400 cpu_manager.go:414] "containerMap: removing container" podUID="30de02c8-b06e-4a94-8185-f30880b602e0" containerName="authorino" Nov 03 00:10:07.590695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.590630 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30de02c8-b06e-4a94-8185-f30880b602e0" containerName="authorino" Nov 03 00:10:07.590864 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.590754 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ace7ca4-13ad-42a0-9881-edc2712e7502" containerName="limitador" Nov 03 00:10:07.590864 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.590767 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="30de02c8-b06e-4a94-8185-f30880b602e0" containerName="authorino" Nov 03 00:10:07.592619 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.592598 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:07.636424 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.636383 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-webhook-cert\") pod \"maas-controller-857dc8657b-4wsbd\" (UID: \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\") " pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:07.636424 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.636429 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8jq\" (UniqueName: \"kubernetes.io/projected/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-kube-api-access-5j8jq\") pod \"maas-controller-857dc8657b-4wsbd\" (UID: \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\") " pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:07.636657 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.636451 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-metrics-certs\") pod \"maas-controller-857dc8657b-4wsbd\" (UID: \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\") " pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:07.646546 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.646515 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 00:10:07.646546 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.646519 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rcpk7\"" Nov 03 00:10:07.646546 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.646521 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 00:10:07.696162 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.696127 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 00:10:07.696162 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.696147 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 00:10:07.696476 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.696228 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 00:10:07.697508 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.697481 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-857dc8657b-4wsbd"] Nov 03 00:10:07.737270 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.737239 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-webhook-cert\") pod \"maas-controller-857dc8657b-4wsbd\" (UID: \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\") " pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:07.737410 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.737274 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8jq\" (UniqueName: \"kubernetes.io/projected/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-kube-api-access-5j8jq\") pod \"maas-controller-857dc8657b-4wsbd\" (UID: \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\") " pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:07.737410 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.737295 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-metrics-certs\") pod \"maas-controller-857dc8657b-4wsbd\" (UID: \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\") " pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:07.739917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.739895 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-webhook-cert\") pod \"maas-controller-857dc8657b-4wsbd\" (UID: \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\") " pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:07.740030 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.739928 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-metrics-certs\") pod \"maas-controller-857dc8657b-4wsbd\" (UID: \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\") " pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:07.792325 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.792284 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:07.793920 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.793889 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8jq\" (UniqueName: \"kubernetes.io/projected/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-kube-api-access-5j8jq\") pod \"maas-controller-857dc8657b-4wsbd\" (UID: \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\") " pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:07.903998 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.903904 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:07.959661 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:07.959626 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:08.186446 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.186367 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-785d4477d7-99w47"] Nov 03 00:10:08.189182 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.189160 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 00:10:08.241446 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.241410 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-metrics-certs\") pod \"maas-controller-785d4477d7-99w47\" (UID: \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\") " pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 00:10:08.241585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.241467 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-webhook-cert\") pod \"maas-controller-785d4477d7-99w47\" (UID: \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\") " pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 00:10:08.241585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.241550 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl22s\" (UniqueName: \"kubernetes.io/projected/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-kube-api-access-hl22s\") pod \"maas-controller-785d4477d7-99w47\" (UID: \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\") " pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 00:10:08.262426 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.262388 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-785d4477d7-99w47"] Nov 03 00:10:08.324763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.324592 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-857dc8657b-4wsbd"] Nov 03 00:10:08.327440 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:10:08.327409 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c82fe1e_1b87_498e_bff8_79a8e961b3ef.slice/crio-a9260280684c960ea7a7817156fc58db20c4fd03bd969d177bdc41d715351185 WatchSource:0}: Error finding container a9260280684c960ea7a7817156fc58db20c4fd03bd969d177bdc41d715351185: Status 404 returned error can't find the container with id a9260280684c960ea7a7817156fc58db20c4fd03bd969d177bdc41d715351185 Nov 03 00:10:08.342718 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.342695 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-webhook-cert\") pod \"maas-controller-785d4477d7-99w47\" (UID: \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\") " pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 00:10:08.342869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.342728 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl22s\" (UniqueName: \"kubernetes.io/projected/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-kube-api-access-hl22s\") pod \"maas-controller-785d4477d7-99w47\" (UID: \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\") " pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 00:10:08.342869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.342782 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-metrics-certs\") pod \"maas-controller-785d4477d7-99w47\" (UID: \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\") " pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 00:10:08.345346 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.345323 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-webhook-cert\") pod \"maas-controller-785d4477d7-99w47\" (UID: \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\") " pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 00:10:08.345431 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.345369 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-metrics-certs\") pod \"maas-controller-785d4477d7-99w47\" (UID: \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\") " pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 00:10:08.399699 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.399672 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl22s\" (UniqueName: \"kubernetes.io/projected/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-kube-api-access-hl22s\") pod \"maas-controller-785d4477d7-99w47\" (UID: \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\") " pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 00:10:08.501207 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.501124 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 00:10:08.874273 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:08.874250 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-785d4477d7-99w47"] Nov 03 00:10:08.876481 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:10:08.876453 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b7d7ab_cb01_490e_98fb_86d5fb539fb7.slice/crio-73052efc5187e9ad0cb315aa0648238a8ef656cd7445aae8e8deba14fabeed2d WatchSource:0}: Error finding container 73052efc5187e9ad0cb315aa0648238a8ef656cd7445aae8e8deba14fabeed2d: Status 404 returned error can't find the container with id 73052efc5187e9ad0cb315aa0648238a8ef656cd7445aae8e8deba14fabeed2d Nov 03 00:10:09.316784 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:09.316731 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785d4477d7-99w47" event={"ID":"13b7d7ab-cb01-490e-98fb-86d5fb539fb7","Type":"ContainerStarted","Data":"73052efc5187e9ad0cb315aa0648238a8ef656cd7445aae8e8deba14fabeed2d"} Nov 03 00:10:09.317913 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:09.317885 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-857dc8657b-4wsbd" event={"ID":"0c82fe1e-1b87-498e-bff8-79a8e961b3ef","Type":"ContainerStarted","Data":"a9260280684c960ea7a7817156fc58db20c4fd03bd969d177bdc41d715351185"} Nov 03 00:10:09.832169 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:09.832133 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:10.031913 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:10.031813 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:12.270962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:12.270883 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:12.338189 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:12.338150 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785d4477d7-99w47" event={"ID":"13b7d7ab-cb01-490e-98fb-86d5fb539fb7","Type":"ContainerStarted","Data":"2fc8dbde268d07914731e6e9e7193ab1ed3c252fe56b8f9a872e2de0083c5b1d"} Nov 03 00:10:12.338377 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:12.338260 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 00:10:12.339724 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:12.339697 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-857dc8657b-4wsbd" event={"ID":"0c82fe1e-1b87-498e-bff8-79a8e961b3ef","Type":"ContainerStarted","Data":"d5cd88f9d527f7fc093bd7ada777652c4f3781a1cd65d6b90baad1667edf35da"} Nov 03 00:10:12.339894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:12.339849 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:12.466501 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:12.466444 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-785d4477d7-99w47" podStartSLOduration=1.42433017 podStartE2EDuration="4.466425373s" totalImagesPullingTime="3.042095203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:12.464352774 +0000 UTC m=+259.415292943" watchObservedRunningTime="2026-11-03 00:10:12.466425373 +0000 UTC m=+259.417365532" Nov 03 00:10:12.470258 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:12.470231 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:12.580391 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:12.580321 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-857dc8657b-4wsbd" podStartSLOduration=2.100175363 podStartE2EDuration="5.5803018s" totalImagesPullingTime="3.480126437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:12.579631104 +0000 UTC m=+259.530571267" watchObservedRunningTime="2026-11-03 00:10:12.5803018 +0000 UTC m=+259.531241959" Nov 03 00:10:15.401479 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:15.401442 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:15.634478 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:15.634442 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:23.345599 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.345566 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 00:10:23.346023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.345640 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:23.513811 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.513772 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-857dc8657b-4wsbd"] Nov 03 00:10:23.514081 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.514025 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-857dc8657b-4wsbd" podUID="0c82fe1e-1b87-498e-bff8-79a8e961b3ef" containerName="manager" containerID="cri-o://d5cd88f9d527f7fc093bd7ada777652c4f3781a1cd65d6b90baad1667edf35da" gracePeriod=30 Nov 03 00:10:23.754476 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.754447 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:23.881725 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.881628 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-webhook-cert\") pod \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\" (UID: \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\") " Nov 03 00:10:23.881725 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.881685 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-metrics-certs\") pod \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\" (UID: \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\") " Nov 03 00:10:23.882020 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.881773 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-kube-api-access-5j8jq\" (UniqueName: \"kubernetes.io/projected/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-kube-api-access-5j8jq\") pod \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\" (UID: \"0c82fe1e-1b87-498e-bff8-79a8e961b3ef\") " Nov 03 00:10:23.885022 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.884987 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-kube-api-access-5j8jq" pod "0c82fe1e-1b87-498e-bff8-79a8e961b3ef" (UID: "0c82fe1e-1b87-498e-bff8-79a8e961b3ef"). InnerVolumeSpecName "kube-api-access-5j8jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:10:23.885173 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.885028 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-webhook-cert" pod "0c82fe1e-1b87-498e-bff8-79a8e961b3ef" (UID: "0c82fe1e-1b87-498e-bff8-79a8e961b3ef"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:10:23.885930 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.885905 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-metrics-certs" pod "0c82fe1e-1b87-498e-bff8-79a8e961b3ef" (UID: "0c82fe1e-1b87-498e-bff8-79a8e961b3ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:10:23.983134 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.983102 6400 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-webhook-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:23.983134 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.983134 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:23.983341 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:23.983145 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j8jq\" (UniqueName: \"kubernetes.io/projected/0c82fe1e-1b87-498e-bff8-79a8e961b3ef-kube-api-access-5j8jq\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:24.391844 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:24.391791 6400 generic.go:356] "Generic (PLEG): container finished" podID="0c82fe1e-1b87-498e-bff8-79a8e961b3ef" containerID="d5cd88f9d527f7fc093bd7ada777652c4f3781a1cd65d6b90baad1667edf35da" exitCode=0 Nov 03 00:10:24.392244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:24.391860 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-857dc8657b-4wsbd" Nov 03 00:10:24.392244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:24.391884 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-857dc8657b-4wsbd" event={"ID":"0c82fe1e-1b87-498e-bff8-79a8e961b3ef","Type":"ContainerDied","Data":"d5cd88f9d527f7fc093bd7ada777652c4f3781a1cd65d6b90baad1667edf35da"} Nov 03 00:10:24.392244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:24.391944 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-857dc8657b-4wsbd" event={"ID":"0c82fe1e-1b87-498e-bff8-79a8e961b3ef","Type":"ContainerDied","Data":"a9260280684c960ea7a7817156fc58db20c4fd03bd969d177bdc41d715351185"} Nov 03 00:10:24.392244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:24.391969 6400 scope.go:122] "RemoveContainer" containerID="d5cd88f9d527f7fc093bd7ada777652c4f3781a1cd65d6b90baad1667edf35da" Nov 03 00:10:24.401509 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:24.401492 6400 scope.go:122] "RemoveContainer" containerID="d5cd88f9d527f7fc093bd7ada777652c4f3781a1cd65d6b90baad1667edf35da" Nov 03 00:10:24.401789 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:10:24.401767 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5cd88f9d527f7fc093bd7ada777652c4f3781a1cd65d6b90baad1667edf35da\": container with ID starting with d5cd88f9d527f7fc093bd7ada777652c4f3781a1cd65d6b90baad1667edf35da not found: ID does not exist" containerID="d5cd88f9d527f7fc093bd7ada777652c4f3781a1cd65d6b90baad1667edf35da" Nov 03 00:10:24.401886 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:24.401799 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5cd88f9d527f7fc093bd7ada777652c4f3781a1cd65d6b90baad1667edf35da"} err="failed to get container status \"d5cd88f9d527f7fc093bd7ada777652c4f3781a1cd65d6b90baad1667edf35da\": rpc error: code = NotFound desc = could not find container \"d5cd88f9d527f7fc093bd7ada777652c4f3781a1cd65d6b90baad1667edf35da\": container with ID starting with d5cd88f9d527f7fc093bd7ada777652c4f3781a1cd65d6b90baad1667edf35da not found: ID does not exist" Nov 03 00:10:24.555391 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:24.555358 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-857dc8657b-4wsbd"] Nov 03 00:10:24.605050 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:24.605023 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-857dc8657b-4wsbd"] Nov 03 00:10:25.178235 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:25.178200 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c82fe1e-1b87-498e-bff8-79a8e961b3ef" path="/var/lib/kubelet/pods/0c82fe1e-1b87-498e-bff8-79a8e961b3ef/volumes" Nov 03 00:10:27.151608 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:10:27.151563 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:10:33.706973 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.704028 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp"] Nov 03 00:10:33.706973 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.704904 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0c82fe1e-1b87-498e-bff8-79a8e961b3ef" containerName="manager" Nov 03 00:10:33.706973 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.704922 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c82fe1e-1b87-498e-bff8-79a8e961b3ef" containerName="manager" Nov 03 00:10:33.706973 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.705099 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c82fe1e-1b87-498e-bff8-79a8e961b3ef" containerName="manager" Nov 03 00:10:33.708919 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.708890 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.768326 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.768284 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/526962f1-ecc2-44db-9485-d25e2082ec97-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.768326 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.768331 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/526962f1-ecc2-44db-9485-d25e2082ec97-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.768545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.768371 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/526962f1-ecc2-44db-9485-d25e2082ec97-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.768545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.768482 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qsrx\" (UniqueName: \"kubernetes.io/projected/526962f1-ecc2-44db-9485-d25e2082ec97-kube-api-access-4qsrx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.768545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.768525 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/526962f1-ecc2-44db-9485-d25e2082ec97-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.768693 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.768548 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/526962f1-ecc2-44db-9485-d25e2082ec97-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.768693 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.768677 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/526962f1-ecc2-44db-9485-d25e2082ec97-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.806515 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.806490 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp"] Nov 03 00:10:33.869770 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.869731 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/526962f1-ecc2-44db-9485-d25e2082ec97-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.869988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.869793 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qsrx\" (UniqueName: \"kubernetes.io/projected/526962f1-ecc2-44db-9485-d25e2082ec97-kube-api-access-4qsrx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.869988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.869840 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/526962f1-ecc2-44db-9485-d25e2082ec97-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.869988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.869862 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/526962f1-ecc2-44db-9485-d25e2082ec97-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.869988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.869905 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/526962f1-ecc2-44db-9485-d25e2082ec97-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.869988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.869930 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/526962f1-ecc2-44db-9485-d25e2082ec97-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.869988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.869953 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/526962f1-ecc2-44db-9485-d25e2082ec97-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.870886 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.870858 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/526962f1-ecc2-44db-9485-d25e2082ec97-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.872322 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.872299 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/526962f1-ecc2-44db-9485-d25e2082ec97-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.872503 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.872485 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/526962f1-ecc2-44db-9485-d25e2082ec97-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.872579 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.872563 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/526962f1-ecc2-44db-9485-d25e2082ec97-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.872618 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.872568 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/526962f1-ecc2-44db-9485-d25e2082ec97-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.927601 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.927573 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qsrx\" (UniqueName: \"kubernetes.io/projected/526962f1-ecc2-44db-9485-d25e2082ec97-kube-api-access-4qsrx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:33.929402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:33.929379 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/526962f1-ecc2-44db-9485-d25e2082ec97-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bcbbp\" (UID: \"526962f1-ecc2-44db-9485-d25e2082ec97\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:34.020928 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:34.020850 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:34.454314 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:34.454289 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp"] Nov 03 00:10:34.456904 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:10:34.456876 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526962f1_ecc2_44db_9485_d25e2082ec97.slice/crio-7d87f36c8f5339fe37ef3f64c6964cb690c97942477bd6090c2bbb0488c898fb WatchSource:0}: Error finding container 7d87f36c8f5339fe37ef3f64c6964cb690c97942477bd6090c2bbb0488c898fb: Status 404 returned error can't find the container with id 7d87f36c8f5339fe37ef3f64c6964cb690c97942477bd6090c2bbb0488c898fb Nov 03 00:10:34.459201 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:34.459168 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:10:34.459276 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:34.459238 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:10:35.446352 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:35.446312 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" event={"ID":"526962f1-ecc2-44db-9485-d25e2082ec97","Type":"ContainerStarted","Data":"db4ef87838e771e992427f67a3039bfe6eb240a72207e3a7cd518355d650ca9a"} Nov 03 00:10:35.446761 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:35.446371 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" event={"ID":"526962f1-ecc2-44db-9485-d25e2082ec97","Type":"ContainerStarted","Data":"7d87f36c8f5339fe37ef3f64c6964cb690c97942477bd6090c2bbb0488c898fb"} Nov 03 00:10:35.446761 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:35.446511 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:35.448087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:35.448063 6400 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bcbbp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:10:35.448225 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:35.448108 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" podUID="526962f1-ecc2-44db-9485-d25e2082ec97" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:10:35.653705 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:35.653650 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" podStartSLOduration=2.653635359 podStartE2EDuration="2.653635359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:35.614232237 +0000 UTC m=+282.565172452" watchObservedRunningTime="2026-11-03 00:10:35.653635359 +0000 UTC m=+282.604575516" Nov 03 00:10:36.451792 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:36.451763 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bcbbp" Nov 03 00:10:36.667894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:36.667862 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2"] Nov 03 00:10:36.668106 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:36.668071 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" podUID="a36b438b-02c6-4d4c-a201-05098c6f7f90" containerName="discovery" containerID="cri-o://90394065e5ca899eda4f80b901c8600bcaeaa1184b1b9413b629eda3d5124dd9" gracePeriod=30 Nov 03 00:10:37.011930 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.011904 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:10:37.099256 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.099222 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-ca-configmap\") pod \"a36b438b-02c6-4d4c-a201-05098c6f7f90\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " Nov 03 00:10:37.099432 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.099302 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-token\" (UniqueName: \"kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-token\") pod \"a36b438b-02c6-4d4c-a201-05098c6f7f90\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " Nov 03 00:10:37.099432 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.099388 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-kubeconfig\") pod \"a36b438b-02c6-4d4c-a201-05098c6f7f90\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " Nov 03 00:10:37.099529 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.099430 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-kube-api-access-sccw4\" (UniqueName: \"kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-kube-api-access-sccw4\") pod \"a36b438b-02c6-4d4c-a201-05098c6f7f90\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " Nov 03 00:10:37.099529 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.099481 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-cacerts\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-cacerts\") pod \"a36b438b-02c6-4d4c-a201-05098c6f7f90\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " Nov 03 00:10:37.099637 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.099540 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-dns-cert\") pod \"a36b438b-02c6-4d4c-a201-05098c6f7f90\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " Nov 03 00:10:37.099637 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.099570 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36b438b-02c6-4d4c-a201-05098c6f7f90-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36b438b-02c6-4d4c-a201-05098c6f7f90-local-certs\") pod \"a36b438b-02c6-4d4c-a201-05098c6f7f90\" (UID: \"a36b438b-02c6-4d4c-a201-05098c6f7f90\") " Nov 03 00:10:37.099991 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.099962 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-ca-configmap" pod "a36b438b-02c6-4d4c-a201-05098c6f7f90" (UID: "a36b438b-02c6-4d4c-a201-05098c6f7f90"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:10:37.102333 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.102295 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-kubeconfig" pod "a36b438b-02c6-4d4c-a201-05098c6f7f90" (UID: "a36b438b-02c6-4d4c-a201-05098c6f7f90"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:10:37.102439 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.102416 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-cacerts" pod "a36b438b-02c6-4d4c-a201-05098c6f7f90" (UID: "a36b438b-02c6-4d4c-a201-05098c6f7f90"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:10:37.102538 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.102515 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36b438b-02c6-4d4c-a201-05098c6f7f90-local-certs" pod "a36b438b-02c6-4d4c-a201-05098c6f7f90" (UID: "a36b438b-02c6-4d4c-a201-05098c6f7f90"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:10:37.102538 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.102526 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-dns-cert" pod "a36b438b-02c6-4d4c-a201-05098c6f7f90" (UID: "a36b438b-02c6-4d4c-a201-05098c6f7f90"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:10:37.102614 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.102543 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-token" pod "a36b438b-02c6-4d4c-a201-05098c6f7f90" (UID: "a36b438b-02c6-4d4c-a201-05098c6f7f90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:10:37.102614 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.102578 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-kube-api-access-sccw4" pod "a36b438b-02c6-4d4c-a201-05098c6f7f90" (UID: "a36b438b-02c6-4d4c-a201-05098c6f7f90"). InnerVolumeSpecName "kube-api-access-sccw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:10:37.200311 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.200284 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:37.200311 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.200310 6400 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-kubeconfig\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:37.200473 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.200340 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sccw4\" (UniqueName: \"kubernetes.io/projected/a36b438b-02c6-4d4c-a201-05098c6f7f90-kube-api-access-sccw4\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:37.200473 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.200349 6400 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-cacerts\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:37.200473 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.200358 6400 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-dns-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:37.200473 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.200366 6400 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36b438b-02c6-4d4c-a201-05098c6f7f90-local-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:37.200473 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.200375 6400 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a36b438b-02c6-4d4c-a201-05098c6f7f90-istio-csr-ca-configmap\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:10:37.456659 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.456564 6400 generic.go:356] "Generic (PLEG): container finished" podID="a36b438b-02c6-4d4c-a201-05098c6f7f90" containerID="90394065e5ca899eda4f80b901c8600bcaeaa1184b1b9413b629eda3d5124dd9" exitCode=0 Nov 03 00:10:37.456659 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.456642 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" Nov 03 00:10:37.457250 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.456651 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" event={"ID":"a36b438b-02c6-4d4c-a201-05098c6f7f90","Type":"ContainerDied","Data":"90394065e5ca899eda4f80b901c8600bcaeaa1184b1b9413b629eda3d5124dd9"} Nov 03 00:10:37.457250 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.456757 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2" event={"ID":"a36b438b-02c6-4d4c-a201-05098c6f7f90","Type":"ContainerDied","Data":"80f6c4d9534df70e6cdfffa804e5063f9934537dbde90614432622ab2177d3fe"} Nov 03 00:10:37.457250 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.456780 6400 scope.go:122] "RemoveContainer" containerID="90394065e5ca899eda4f80b901c8600bcaeaa1184b1b9413b629eda3d5124dd9" Nov 03 00:10:37.465363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.465340 6400 scope.go:122] "RemoveContainer" containerID="90394065e5ca899eda4f80b901c8600bcaeaa1184b1b9413b629eda3d5124dd9" Nov 03 00:10:37.465667 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:10:37.465644 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90394065e5ca899eda4f80b901c8600bcaeaa1184b1b9413b629eda3d5124dd9\": container with ID starting with 90394065e5ca899eda4f80b901c8600bcaeaa1184b1b9413b629eda3d5124dd9 not found: ID does not exist" containerID="90394065e5ca899eda4f80b901c8600bcaeaa1184b1b9413b629eda3d5124dd9" Nov 03 00:10:37.465751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.465672 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90394065e5ca899eda4f80b901c8600bcaeaa1184b1b9413b629eda3d5124dd9"} err="failed to get container status \"90394065e5ca899eda4f80b901c8600bcaeaa1184b1b9413b629eda3d5124dd9\": rpc error: code = NotFound desc = could not find container \"90394065e5ca899eda4f80b901c8600bcaeaa1184b1b9413b629eda3d5124dd9\": container with ID starting with 90394065e5ca899eda4f80b901c8600bcaeaa1184b1b9413b629eda3d5124dd9 not found: ID does not exist" Nov 03 00:10:37.621484 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.621459 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2"] Nov 03 00:10:37.671017 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:37.670988 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-stdh2"] Nov 03 00:10:38.219143 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:38.219111 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:38.378456 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:38.378420 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:38.565584 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:38.565548 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:38.773957 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:38.773927 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:38.932642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:38.932559 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:39.033537 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.033504 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-62pxm"] Nov 03 00:10:39.033949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.033935 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a36b438b-02c6-4d4c-a201-05098c6f7f90" containerName="discovery" Nov 03 00:10:39.034004 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.033951 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a36b438b-02c6-4d4c-a201-05098c6f7f90" containerName="discovery" Nov 03 00:10:39.034035 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.034013 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a36b438b-02c6-4d4c-a201-05098c6f7f90" containerName="discovery" Nov 03 00:10:39.038331 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.038312 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-62pxm" Nov 03 00:10:39.091645 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.091613 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-br25w\"" Nov 03 00:10:39.091861 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.091715 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 00:10:39.093305 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.093282 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-62pxm"] Nov 03 00:10:39.093596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.093571 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:39.117777 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.117751 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1-cert\") pod \"kserve-controller-manager-776cc66d47-62pxm\" (UID: \"288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-62pxm" Nov 03 00:10:39.117933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.117798 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk67q\" (UniqueName: \"kubernetes.io/projected/288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1-kube-api-access-fk67q\") pod \"kserve-controller-manager-776cc66d47-62pxm\" (UID: \"288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-62pxm" Nov 03 00:10:39.146674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.146645 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc"] Nov 03 00:10:39.150112 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.150097 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" Nov 03 00:10:39.177017 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.176986 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a36b438b-02c6-4d4c-a201-05098c6f7f90" path="/var/lib/kubelet/pods/a36b438b-02c6-4d4c-a201-05098c6f7f90/volumes" Nov 03 00:10:39.202743 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.202684 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-qmgdp\"" Nov 03 00:10:39.202743 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.202685 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 00:10:39.207971 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.207947 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc"] Nov 03 00:10:39.218512 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.218490 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sw2dc\" (UID: \"0e874e2e-b7f2-48f2-b538-7632ed57ecb0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" Nov 03 00:10:39.218640 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.218530 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1-cert\") pod \"kserve-controller-manager-776cc66d47-62pxm\" (UID: \"288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-62pxm" Nov 03 00:10:39.218640 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.218567 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk67q\" (UniqueName: \"kubernetes.io/projected/288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1-kube-api-access-fk67q\") pod \"kserve-controller-manager-776cc66d47-62pxm\" (UID: \"288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-62pxm" Nov 03 00:10:39.218739 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.218649 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5p2b\" (UniqueName: \"kubernetes.io/projected/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-kube-api-access-r5p2b\") pod \"llmisvc-controller-manager-869d5dcc5d-sw2dc\" (UID: \"0e874e2e-b7f2-48f2-b538-7632ed57ecb0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" Nov 03 00:10:39.221095 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.221074 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1-cert\") pod \"kserve-controller-manager-776cc66d47-62pxm\" (UID: \"288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-62pxm" Nov 03 00:10:39.259663 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.259638 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:39.277382 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.277356 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk67q\" (UniqueName: \"kubernetes.io/projected/288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1-kube-api-access-fk67q\") pod \"kserve-controller-manager-776cc66d47-62pxm\" (UID: \"288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-62pxm" Nov 03 00:10:39.319275 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.319251 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5p2b\" (UniqueName: \"kubernetes.io/projected/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-kube-api-access-r5p2b\") pod \"llmisvc-controller-manager-869d5dcc5d-sw2dc\" (UID: \"0e874e2e-b7f2-48f2-b538-7632ed57ecb0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" Nov 03 00:10:39.319419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.319308 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sw2dc\" (UID: \"0e874e2e-b7f2-48f2-b538-7632ed57ecb0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" Nov 03 00:10:39.321894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.321871 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sw2dc\" (UID: \"0e874e2e-b7f2-48f2-b538-7632ed57ecb0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" Nov 03 00:10:39.351775 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.351738 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-62pxm" Nov 03 00:10:39.376230 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.376203 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5p2b\" (UniqueName: \"kubernetes.io/projected/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-kube-api-access-r5p2b\") pod \"llmisvc-controller-manager-869d5dcc5d-sw2dc\" (UID: \"0e874e2e-b7f2-48f2-b538-7632ed57ecb0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" Nov 03 00:10:39.462475 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.462396 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" Nov 03 00:10:39.723402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.723193 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-62pxm"] Nov 03 00:10:39.726073 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:10:39.726040 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288ffc8d_dc5b_4bd0_a9ef_bb45712f56a1.slice/crio-4e58353a3267f20c65fcda5d0657002148ad693b0ab2b559f9e253f6ebecdb0c WatchSource:0}: Error finding container 4e58353a3267f20c65fcda5d0657002148ad693b0ab2b559f9e253f6ebecdb0c: Status 404 returned error can't find the container with id 4e58353a3267f20c65fcda5d0657002148ad693b0ab2b559f9e253f6ebecdb0c Nov 03 00:10:39.831741 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:39.831718 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc"] Nov 03 00:10:39.834093 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:10:39.834065 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0e874e2e_b7f2_48f2_b538_7632ed57ecb0.slice/crio-b7a3f4093ab543773489498e3f27327e9f2604c58a57496ab1cf6872cc24349a WatchSource:0}: Error finding container b7a3f4093ab543773489498e3f27327e9f2604c58a57496ab1cf6872cc24349a: Status 404 returned error can't find the container with id b7a3f4093ab543773489498e3f27327e9f2604c58a57496ab1cf6872cc24349a Nov 03 00:10:40.409963 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:40.409928 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:40.474954 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:40.474916 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-62pxm" event={"ID":"288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1","Type":"ContainerStarted","Data":"4e58353a3267f20c65fcda5d0657002148ad693b0ab2b559f9e253f6ebecdb0c"} Nov 03 00:10:40.476144 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:40.476117 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" event={"ID":"0e874e2e-b7f2-48f2-b538-7632ed57ecb0","Type":"ContainerStarted","Data":"b7a3f4093ab543773489498e3f27327e9f2604c58a57496ab1cf6872cc24349a"} Nov 03 00:10:43.495513 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:43.495485 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-62pxm" event={"ID":"288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1","Type":"ContainerStarted","Data":"4e75b7a87536f35f0d528481a67403b576b5d89d881b1b758f1609b5ed06379d"} Nov 03 00:10:43.495874 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:43.495606 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-62pxm" Nov 03 00:10:43.497156 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:43.497142 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" Nov 03 00:10:43.611147 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:43.611059 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-62pxm" podStartSLOduration=2.023141723 podStartE2EDuration="5.611045206s" totalImagesPullingTime="3.587903483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:43.610667798 +0000 UTC m=+290.561607955" watchObservedRunningTime="2026-11-03 00:10:43.611045206 +0000 UTC m=+290.561985361" Nov 03 00:10:43.728126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:43.728071 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" podStartSLOduration=1.234208625 podStartE2EDuration="4.728051998s" totalImagesPullingTime="3.493843373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:43.727259342 +0000 UTC m=+290.678199502" watchObservedRunningTime="2026-11-03 00:10:43.728051998 +0000 UTC m=+290.678992153" Nov 03 00:10:44.503498 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:44.503462 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" event={"ID":"0e874e2e-b7f2-48f2-b538-7632ed57ecb0","Type":"ContainerStarted","Data":"0ca6aabb0fe797058398a9b49de39d51282d36118b94012efaf427745bfeb6b0"} Nov 03 00:10:45.373030 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:45.372997 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:45.645333 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:45.645245 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:45.799643 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:45.799594 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:45.963023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:45.962927 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:46.153799 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:46.153755 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:46.390606 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:46.390567 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:46.545192 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:46.545162 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:47.524708 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:47.524664 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:47.754318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:47.754284 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:47.906139 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:47.906088 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:48.069949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.069902 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:48.673722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.673690 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq"] Nov 03 00:10:48.678464 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.678446 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:48.779982 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.779949 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:10:48.779982 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.779948 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:10:48.780225 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.780085 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 00:10:48.780284 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.780239 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kgpd5\"" Nov 03 00:10:48.780419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.780397 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:10:48.781772 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.781752 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq"] Nov 03 00:10:48.812789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.812759 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18531534-0eff-444b-b11d-61cfc563811d-openshift-service-ca\") pod \"maas-api-64d6cbf9dd-gg6nq\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:48.812953 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.812805 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-metrics-certs\") pod \"maas-api-64d6cbf9dd-gg6nq\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:48.812953 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.812843 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54c8\" (UniqueName: \"kubernetes.io/projected/18531534-0eff-444b-b11d-61cfc563811d-kube-api-access-g54c8\") pod \"maas-api-64d6cbf9dd-gg6nq\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:48.813064 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.813042 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-maas-api-tls\") pod \"maas-api-64d6cbf9dd-gg6nq\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:48.909980 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.909949 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc"] Nov 03 00:10:48.912283 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.912263 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:48.913573 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.913553 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-maas-api-tls\") pod \"maas-api-64d6cbf9dd-gg6nq\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:48.913655 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.913598 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18531534-0eff-444b-b11d-61cfc563811d-openshift-service-ca\") pod \"maas-api-64d6cbf9dd-gg6nq\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:48.913655 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.913644 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-metrics-certs\") pod \"maas-api-64d6cbf9dd-gg6nq\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:48.913750 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.913668 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54c8\" (UniqueName: \"kubernetes.io/projected/18531534-0eff-444b-b11d-61cfc563811d-kube-api-access-g54c8\") pod \"maas-api-64d6cbf9dd-gg6nq\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:48.914354 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.914333 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18531534-0eff-444b-b11d-61cfc563811d-openshift-service-ca\") pod \"maas-api-64d6cbf9dd-gg6nq\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:48.916282 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.916261 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-metrics-certs\") pod \"maas-api-64d6cbf9dd-gg6nq\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:48.916354 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.916298 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-maas-api-tls\") pod \"maas-api-64d6cbf9dd-gg6nq\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:48.964363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.964255 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 00:10:48.964363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.964319 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:10:48.964363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.964345 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 00:10:48.964363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.964345 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-xkwkt\"" Nov 03 00:10:48.964656 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.964374 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 00:10:48.964656 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.964384 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 00:10:48.969590 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.969569 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc"] Nov 03 00:10:48.975112 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.975091 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54c8\" (UniqueName: \"kubernetes.io/projected/18531534-0eff-444b-b11d-61cfc563811d-kube-api-access-g54c8\") pod \"maas-api-64d6cbf9dd-gg6nq\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:48.990921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:48.990891 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:49.015020 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.014997 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b-metrics-certs\") pod \"jobset-controller-manager-55b4954d7b-t96mc\" (UID: \"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b\") " pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:49.015148 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.015026 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb6rr\" (UniqueName: \"kubernetes.io/projected/8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b-kube-api-access-mb6rr\") pod \"jobset-controller-manager-55b4954d7b-t96mc\" (UID: \"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b\") " pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:49.015148 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.015051 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b-cert\") pod \"jobset-controller-manager-55b4954d7b-t96mc\" (UID: \"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b\") " pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:49.015231 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.015174 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b-manager-config\") pod \"jobset-controller-manager-55b4954d7b-t96mc\" (UID: \"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b\") " pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:49.116601 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.116567 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b-manager-config\") pod \"jobset-controller-manager-55b4954d7b-t96mc\" (UID: \"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b\") " pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:49.116840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.116690 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b-metrics-certs\") pod \"jobset-controller-manager-55b4954d7b-t96mc\" (UID: \"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b\") " pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:49.116840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.116722 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb6rr\" (UniqueName: \"kubernetes.io/projected/8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b-kube-api-access-mb6rr\") pod \"jobset-controller-manager-55b4954d7b-t96mc\" (UID: \"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b\") " pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:49.116840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.116776 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b-cert\") pod \"jobset-controller-manager-55b4954d7b-t96mc\" (UID: \"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b\") " pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:49.117377 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.117347 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b-manager-config\") pod \"jobset-controller-manager-55b4954d7b-t96mc\" (UID: \"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b\") " pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:49.119368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.119350 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b-cert\") pod \"jobset-controller-manager-55b4954d7b-t96mc\" (UID: \"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b\") " pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:49.119534 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.119514 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b-metrics-certs\") pod \"jobset-controller-manager-55b4954d7b-t96mc\" (UID: \"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b\") " pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:49.173261 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.173228 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb6rr\" (UniqueName: \"kubernetes.io/projected/8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b-kube-api-access-mb6rr\") pod \"jobset-controller-manager-55b4954d7b-t96mc\" (UID: \"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b\") " pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:49.232257 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.232185 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:49.414956 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.414931 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq"] Nov 03 00:10:49.417374 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:10:49.417349 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18531534_0eff_444b_b11d_61cfc563811d.slice/crio-53df5de2b5b217a71377650de1fe8cef38a580b6894f6afb9b2eb40a1b415091 WatchSource:0}: Error finding container 53df5de2b5b217a71377650de1fe8cef38a580b6894f6afb9b2eb40a1b415091: Status 404 returned error can't find the container with id 53df5de2b5b217a71377650de1fe8cef38a580b6894f6afb9b2eb40a1b415091 Nov 03 00:10:49.527320 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.527246 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" event={"ID":"18531534-0eff-444b-b11d-61cfc563811d","Type":"ContainerStarted","Data":"53df5de2b5b217a71377650de1fe8cef38a580b6894f6afb9b2eb40a1b415091"} Nov 03 00:10:49.600225 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:49.600196 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc"] Nov 03 00:10:49.603421 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:10:49.603391 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9be2d9_3e96_4ffe_ab8d_98bf9bed504b.slice/crio-04dd9f2a401af205cedadd8e1e7e24dbaa90edece32d9c23c95cd88a498d39df WatchSource:0}: Error finding container 04dd9f2a401af205cedadd8e1e7e24dbaa90edece32d9c23c95cd88a498d39df: Status 404 returned error can't find the container with id 04dd9f2a401af205cedadd8e1e7e24dbaa90edece32d9c23c95cd88a498d39df Nov 03 00:10:50.533168 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:50.533129 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" event={"ID":"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b","Type":"ContainerStarted","Data":"04dd9f2a401af205cedadd8e1e7e24dbaa90edece32d9c23c95cd88a498d39df"} Nov 03 00:10:53.487803 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:53.484422 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:10:54.555492 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:54.555455 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" event={"ID":"8d9be2d9-3e96-4ffe-ab8d-98bf9bed504b","Type":"ContainerStarted","Data":"41d639d5efd55e0245fd6995938e87c527c26365d4e4d2617701941747b10ee4"} Nov 03 00:10:54.555969 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:54.555663 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:10:54.668183 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:54.668131 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" podStartSLOduration=2.5785437289999997 podStartE2EDuration="6.668116521s" totalImagesPullingTime="4.089572792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:54.667260386 +0000 UTC m=+301.618200546" watchObservedRunningTime="2026-11-03 00:10:54.668116521 +0000 UTC m=+301.619056677" Nov 03 00:10:56.565984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:56.565951 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" event={"ID":"18531534-0eff-444b-b11d-61cfc563811d","Type":"ContainerStarted","Data":"ffba9c3005f887ebe21e70dc84506d104c51d089aaab342c34ed7e17740b6f99"} Nov 03 00:10:56.566375 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:56.566100 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:10:56.685839 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:56.685780 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" podStartSLOduration=1.904678224 podStartE2EDuration="8.68576582s" totalImagesPullingTime="6.781087596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:56.683938118 +0000 UTC m=+303.634878275" watchObservedRunningTime="2026-11-03 00:10:56.68576582 +0000 UTC m=+303.636705977" Nov 03 00:10:57.151861 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:10:57.151799 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:10:59.786749 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:10:59.786713 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:02.575838 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:02.575801 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:11:03.149787 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:03.149753 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:03.423094 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:03.423019 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:03.756155 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:03.756074 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cdcfc9dd5-h7cdk"] Nov 03 00:11:03.758630 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:03.758609 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cdcfc9dd5-h7cdk" Nov 03 00:11:03.814086 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:03.814050 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cdcfc9dd5-h7cdk"] Nov 03 00:11:03.856231 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:03.856200 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn8g8\" (UniqueName: \"kubernetes.io/projected/702a0068-f89c-49b7-a675-022607ad46b9-kube-api-access-dn8g8\") pod \"authorino-6cdcfc9dd5-h7cdk\" (UID: \"702a0068-f89c-49b7-a675-022607ad46b9\") " pod="kuadrant-system/authorino-6cdcfc9dd5-h7cdk" Nov 03 00:11:03.957048 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:03.957011 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn8g8\" (UniqueName: \"kubernetes.io/projected/702a0068-f89c-49b7-a675-022607ad46b9-kube-api-access-dn8g8\") pod \"authorino-6cdcfc9dd5-h7cdk\" (UID: \"702a0068-f89c-49b7-a675-022607ad46b9\") " pod="kuadrant-system/authorino-6cdcfc9dd5-h7cdk" Nov 03 00:11:04.019369 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.019297 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn8g8\" (UniqueName: \"kubernetes.io/projected/702a0068-f89c-49b7-a675-022607ad46b9-kube-api-access-dn8g8\") pod \"authorino-6cdcfc9dd5-h7cdk\" (UID: \"702a0068-f89c-49b7-a675-022607ad46b9\") " pod="kuadrant-system/authorino-6cdcfc9dd5-h7cdk" Nov 03 00:11:04.069397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.069365 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cdcfc9dd5-h7cdk" Nov 03 00:11:04.219221 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.219186 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cdcfc9dd5-h7cdk"] Nov 03 00:11:04.243361 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.243333 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69cd468fff-k2b5p"] Nov 03 00:11:04.245799 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.245783 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69cd468fff-k2b5p" Nov 03 00:11:04.259862 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.259813 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6tj2\" (UniqueName: \"kubernetes.io/projected/421a7e2c-a20f-45f8-a213-e338caf92496-kube-api-access-l6tj2\") pod \"authorino-69cd468fff-k2b5p\" (UID: \"421a7e2c-a20f-45f8-a213-e338caf92496\") " pod="kuadrant-system/authorino-69cd468fff-k2b5p" Nov 03 00:11:04.260052 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.260032 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/421a7e2c-a20f-45f8-a213-e338caf92496-tls-cert\") pod \"authorino-69cd468fff-k2b5p\" (UID: \"421a7e2c-a20f-45f8-a213-e338caf92496\") " pod="kuadrant-system/authorino-69cd468fff-k2b5p" Nov 03 00:11:04.300392 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.300366 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 00:11:04.302051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.302026 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69cd468fff-k2b5p"] Nov 03 00:11:04.366406 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.361155 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6tj2\" (UniqueName: \"kubernetes.io/projected/421a7e2c-a20f-45f8-a213-e338caf92496-kube-api-access-l6tj2\") pod \"authorino-69cd468fff-k2b5p\" (UID: \"421a7e2c-a20f-45f8-a213-e338caf92496\") " pod="kuadrant-system/authorino-69cd468fff-k2b5p" Nov 03 00:11:04.366406 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.361297 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/421a7e2c-a20f-45f8-a213-e338caf92496-tls-cert\") pod \"authorino-69cd468fff-k2b5p\" (UID: \"421a7e2c-a20f-45f8-a213-e338caf92496\") " pod="kuadrant-system/authorino-69cd468fff-k2b5p" Nov 03 00:11:04.367184 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.367151 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/421a7e2c-a20f-45f8-a213-e338caf92496-tls-cert\") pod \"authorino-69cd468fff-k2b5p\" (UID: \"421a7e2c-a20f-45f8-a213-e338caf92496\") " pod="kuadrant-system/authorino-69cd468fff-k2b5p" Nov 03 00:11:04.419435 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.419409 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6tj2\" (UniqueName: \"kubernetes.io/projected/421a7e2c-a20f-45f8-a213-e338caf92496-kube-api-access-l6tj2\") pod \"authorino-69cd468fff-k2b5p\" (UID: \"421a7e2c-a20f-45f8-a213-e338caf92496\") " pod="kuadrant-system/authorino-69cd468fff-k2b5p" Nov 03 00:11:04.441448 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.441424 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cdcfc9dd5-h7cdk"] Nov 03 00:11:04.444028 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:11:04.444002 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod702a0068_f89c_49b7_a675_022607ad46b9.slice/crio-53822b8212c4ec148023ec8d9ff151c2a6fa72e29d4ff108b87720e5548d040d WatchSource:0}: Error finding container 53822b8212c4ec148023ec8d9ff151c2a6fa72e29d4ff108b87720e5548d040d: Status 404 returned error can't find the container with id 53822b8212c4ec148023ec8d9ff151c2a6fa72e29d4ff108b87720e5548d040d Nov 03 00:11:04.557609 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.557504 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69cd468fff-k2b5p" Nov 03 00:11:04.602869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.602815 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cdcfc9dd5-h7cdk" event={"ID":"702a0068-f89c-49b7-a675-022607ad46b9","Type":"ContainerStarted","Data":"53822b8212c4ec148023ec8d9ff151c2a6fa72e29d4ff108b87720e5548d040d"} Nov 03 00:11:04.927160 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:04.927137 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69cd468fff-k2b5p"] Nov 03 00:11:04.929712 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:11:04.929682 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod421a7e2c_a20f_45f8_a213_e338caf92496.slice/crio-dfcf85006a4ef56ad3d37b3873c8bb3b2aeffdc0b0aafad528d9fdce4ef39550 WatchSource:0}: Error finding container dfcf85006a4ef56ad3d37b3873c8bb3b2aeffdc0b0aafad528d9fdce4ef39550: Status 404 returned error can't find the container with id dfcf85006a4ef56ad3d37b3873c8bb3b2aeffdc0b0aafad528d9fdce4ef39550 Nov 03 00:11:05.564716 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:05.564679 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-55b4954d7b-t96mc" Nov 03 00:11:05.608836 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:05.608784 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69cd468fff-k2b5p" event={"ID":"421a7e2c-a20f-45f8-a213-e338caf92496","Type":"ContainerStarted","Data":"dfcf85006a4ef56ad3d37b3873c8bb3b2aeffdc0b0aafad528d9fdce4ef39550"} Nov 03 00:11:06.615135 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:06.615087 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69cd468fff-k2b5p" event={"ID":"421a7e2c-a20f-45f8-a213-e338caf92496","Type":"ContainerStarted","Data":"0808dce92ecbcae6a0ee76b3d770e3745c5423cc49bdbf4acaaaa65b4ec2b8e8"} Nov 03 00:11:06.616655 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:06.616624 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cdcfc9dd5-h7cdk" event={"ID":"702a0068-f89c-49b7-a675-022607ad46b9","Type":"ContainerStarted","Data":"4f407021de37f8822d44d94334ded6b3c92e89b393f0030af1ec81d7d148bcd9"} Nov 03 00:11:06.616853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:06.616684 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6cdcfc9dd5-h7cdk" podUID="702a0068-f89c-49b7-a675-022607ad46b9" containerName="authorino" containerID="cri-o://4f407021de37f8822d44d94334ded6b3c92e89b393f0030af1ec81d7d148bcd9" gracePeriod=30 Nov 03 00:11:06.727969 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:06.727646 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-69cd468fff-k2b5p" podStartSLOduration=1.257701472 podStartE2EDuration="2.727629083s" totalImagesPullingTime="1.469927611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:06.72643166 +0000 UTC m=+313.677371822" watchObservedRunningTime="2026-11-03 00:11:06.727629083 +0000 UTC m=+313.678569270" Nov 03 00:11:06.751345 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:06.751309 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-p5fb4"] Nov 03 00:11:06.751503 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:06.751489 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-p5fb4" podUID="9cb77863-cd9c-418b-91e3-96ceec1fd8fc" containerName="authorino" containerID="cri-o://59eb60cf6b64aa5d655d1536c10f6a5a5769e8ee3335c735cdd95e8ef4196434" gracePeriod=30 Nov 03 00:11:06.837219 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:06.837167 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6cdcfc9dd5-h7cdk" podStartSLOduration=2.545133061 podStartE2EDuration="3.837152714s" totalImagesPullingTime="1.292019653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:06.836913915 +0000 UTC m=+313.787854073" watchObservedRunningTime="2026-11-03 00:11:06.837152714 +0000 UTC m=+313.788092871" Nov 03 00:11:06.865081 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:06.865056 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cdcfc9dd5-h7cdk" Nov 03 00:11:06.884895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:06.884862 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702a0068-f89c-49b7-a675-022607ad46b9-kube-api-access-dn8g8\" (UniqueName: \"kubernetes.io/projected/702a0068-f89c-49b7-a675-022607ad46b9-kube-api-access-dn8g8\") pod \"702a0068-f89c-49b7-a675-022607ad46b9\" (UID: \"702a0068-f89c-49b7-a675-022607ad46b9\") " Nov 03 00:11:06.886936 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:06.886910 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702a0068-f89c-49b7-a675-022607ad46b9-kube-api-access-dn8g8" pod "702a0068-f89c-49b7-a675-022607ad46b9" (UID: "702a0068-f89c-49b7-a675-022607ad46b9"). InnerVolumeSpecName "kube-api-access-dn8g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:06.985917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:06.985878 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn8g8\" (UniqueName: \"kubernetes.io/projected/702a0068-f89c-49b7-a675-022607ad46b9-kube-api-access-dn8g8\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:06.990837 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:06.990798 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-p5fb4" Nov 03 00:11:07.086814 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.086782 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb77863-cd9c-418b-91e3-96ceec1fd8fc-kube-api-access-2v57k\" (UniqueName: \"kubernetes.io/projected/9cb77863-cd9c-418b-91e3-96ceec1fd8fc-kube-api-access-2v57k\") pod \"9cb77863-cd9c-418b-91e3-96ceec1fd8fc\" (UID: \"9cb77863-cd9c-418b-91e3-96ceec1fd8fc\") " Nov 03 00:11:07.089011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.088979 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb77863-cd9c-418b-91e3-96ceec1fd8fc-kube-api-access-2v57k" pod "9cb77863-cd9c-418b-91e3-96ceec1fd8fc" (UID: "9cb77863-cd9c-418b-91e3-96ceec1fd8fc"). InnerVolumeSpecName "kube-api-access-2v57k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:07.189417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.189387 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v57k\" (UniqueName: \"kubernetes.io/projected/9cb77863-cd9c-418b-91e3-96ceec1fd8fc-kube-api-access-2v57k\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:07.623175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.623138 6400 generic.go:356] "Generic (PLEG): container finished" podID="702a0068-f89c-49b7-a675-022607ad46b9" containerID="4f407021de37f8822d44d94334ded6b3c92e89b393f0030af1ec81d7d148bcd9" exitCode=0 Nov 03 00:11:07.623659 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.623202 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cdcfc9dd5-h7cdk" Nov 03 00:11:07.623659 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.623235 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cdcfc9dd5-h7cdk" event={"ID":"702a0068-f89c-49b7-a675-022607ad46b9","Type":"ContainerDied","Data":"4f407021de37f8822d44d94334ded6b3c92e89b393f0030af1ec81d7d148bcd9"} Nov 03 00:11:07.623659 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.623275 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cdcfc9dd5-h7cdk" event={"ID":"702a0068-f89c-49b7-a675-022607ad46b9","Type":"ContainerDied","Data":"53822b8212c4ec148023ec8d9ff151c2a6fa72e29d4ff108b87720e5548d040d"} Nov 03 00:11:07.623659 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.623298 6400 scope.go:122] "RemoveContainer" containerID="4f407021de37f8822d44d94334ded6b3c92e89b393f0030af1ec81d7d148bcd9" Nov 03 00:11:07.625367 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.625342 6400 generic.go:356] "Generic (PLEG): container finished" podID="9cb77863-cd9c-418b-91e3-96ceec1fd8fc" containerID="59eb60cf6b64aa5d655d1536c10f6a5a5769e8ee3335c735cdd95e8ef4196434" exitCode=0 Nov 03 00:11:07.625491 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.625463 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-p5fb4" Nov 03 00:11:07.625555 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.625513 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-p5fb4" event={"ID":"9cb77863-cd9c-418b-91e3-96ceec1fd8fc","Type":"ContainerDied","Data":"59eb60cf6b64aa5d655d1536c10f6a5a5769e8ee3335c735cdd95e8ef4196434"} Nov 03 00:11:07.625555 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.625547 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-p5fb4" event={"ID":"9cb77863-cd9c-418b-91e3-96ceec1fd8fc","Type":"ContainerDied","Data":"b906b13783da032d9cceec7a97d359a7d36c1195ce5f2a41d428df8bdbf499bd"} Nov 03 00:11:07.635602 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.635583 6400 scope.go:122] "RemoveContainer" containerID="4f407021de37f8822d44d94334ded6b3c92e89b393f0030af1ec81d7d148bcd9" Nov 03 00:11:07.635901 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:11:07.635886 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f407021de37f8822d44d94334ded6b3c92e89b393f0030af1ec81d7d148bcd9\": container with ID starting with 4f407021de37f8822d44d94334ded6b3c92e89b393f0030af1ec81d7d148bcd9 not found: ID does not exist" containerID="4f407021de37f8822d44d94334ded6b3c92e89b393f0030af1ec81d7d148bcd9" Nov 03 00:11:07.635955 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.635911 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f407021de37f8822d44d94334ded6b3c92e89b393f0030af1ec81d7d148bcd9"} err="failed to get container status \"4f407021de37f8822d44d94334ded6b3c92e89b393f0030af1ec81d7d148bcd9\": rpc error: code = NotFound desc = could not find container \"4f407021de37f8822d44d94334ded6b3c92e89b393f0030af1ec81d7d148bcd9\": container with ID starting with 4f407021de37f8822d44d94334ded6b3c92e89b393f0030af1ec81d7d148bcd9 not found: ID does not exist" Nov 03 00:11:07.635955 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.635927 6400 scope.go:122] "RemoveContainer" containerID="59eb60cf6b64aa5d655d1536c10f6a5a5769e8ee3335c735cdd95e8ef4196434" Nov 03 00:11:07.643580 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.643556 6400 scope.go:122] "RemoveContainer" containerID="59eb60cf6b64aa5d655d1536c10f6a5a5769e8ee3335c735cdd95e8ef4196434" Nov 03 00:11:07.643862 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:11:07.643845 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59eb60cf6b64aa5d655d1536c10f6a5a5769e8ee3335c735cdd95e8ef4196434\": container with ID starting with 59eb60cf6b64aa5d655d1536c10f6a5a5769e8ee3335c735cdd95e8ef4196434 not found: ID does not exist" containerID="59eb60cf6b64aa5d655d1536c10f6a5a5769e8ee3335c735cdd95e8ef4196434" Nov 03 00:11:07.643919 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.643869 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59eb60cf6b64aa5d655d1536c10f6a5a5769e8ee3335c735cdd95e8ef4196434"} err="failed to get container status \"59eb60cf6b64aa5d655d1536c10f6a5a5769e8ee3335c735cdd95e8ef4196434\": rpc error: code = NotFound desc = could not find container \"59eb60cf6b64aa5d655d1536c10f6a5a5769e8ee3335c735cdd95e8ef4196434\": container with ID starting with 59eb60cf6b64aa5d655d1536c10f6a5a5769e8ee3335c735cdd95e8ef4196434 not found: ID does not exist" Nov 03 00:11:07.795883 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.795838 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cdcfc9dd5-h7cdk"] Nov 03 00:11:07.798582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.798547 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cdcfc9dd5-h7cdk"] Nov 03 00:11:07.906838 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.906797 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-p5fb4"] Nov 03 00:11:07.907894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:07.907872 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-p5fb4"] Nov 03 00:11:09.178391 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:09.178355 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702a0068-f89c-49b7-a675-022607ad46b9" path="/var/lib/kubelet/pods/702a0068-f89c-49b7-a675-022607ad46b9/volumes" Nov 03 00:11:09.178748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:09.178677 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb77863-cd9c-418b-91e3-96ceec1fd8fc" path="/var/lib/kubelet/pods/9cb77863-cd9c-418b-91e3-96ceec1fd8fc/volumes" Nov 03 00:11:09.452624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:09.452541 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:09.644180 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:09.644143 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:13.544834 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.544794 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq"] Nov 03 00:11:13.545279 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.545016 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" podUID="18531534-0eff-444b-b11d-61cfc563811d" containerName="maas-api" containerID="cri-o://ffba9c3005f887ebe21e70dc84506d104c51d089aaab342c34ed7e17740b6f99" gracePeriod=30 Nov 03 00:11:13.787514 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.787473 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:11:13.852752 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.852651 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-metrics-certs\") pod \"18531534-0eff-444b-b11d-61cfc563811d\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " Nov 03 00:11:13.852752 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.852689 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18531534-0eff-444b-b11d-61cfc563811d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18531534-0eff-444b-b11d-61cfc563811d-openshift-service-ca\") pod \"18531534-0eff-444b-b11d-61cfc563811d\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " Nov 03 00:11:13.852752 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.852723 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18531534-0eff-444b-b11d-61cfc563811d-kube-api-access-g54c8\" (UniqueName: \"kubernetes.io/projected/18531534-0eff-444b-b11d-61cfc563811d-kube-api-access-g54c8\") pod \"18531534-0eff-444b-b11d-61cfc563811d\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " Nov 03 00:11:13.853054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.852852 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-maas-api-tls\") pod \"18531534-0eff-444b-b11d-61cfc563811d\" (UID: \"18531534-0eff-444b-b11d-61cfc563811d\") " Nov 03 00:11:13.853113 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.853094 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18531534-0eff-444b-b11d-61cfc563811d-openshift-service-ca" pod "18531534-0eff-444b-b11d-61cfc563811d" (UID: "18531534-0eff-444b-b11d-61cfc563811d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:11:13.855223 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.855191 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-metrics-certs" pod "18531534-0eff-444b-b11d-61cfc563811d" (UID: "18531534-0eff-444b-b11d-61cfc563811d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:11:13.855332 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.855236 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18531534-0eff-444b-b11d-61cfc563811d-kube-api-access-g54c8" pod "18531534-0eff-444b-b11d-61cfc563811d" (UID: "18531534-0eff-444b-b11d-61cfc563811d"). InnerVolumeSpecName "kube-api-access-g54c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:13.855332 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.855256 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-maas-api-tls" pod "18531534-0eff-444b-b11d-61cfc563811d" (UID: "18531534-0eff-444b-b11d-61cfc563811d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:11:13.953728 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.953690 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:13.953728 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.953722 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18531534-0eff-444b-b11d-61cfc563811d-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:13.953977 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.953738 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18531534-0eff-444b-b11d-61cfc563811d-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:13.953977 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:13.953755 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g54c8\" (UniqueName: \"kubernetes.io/projected/18531534-0eff-444b-b11d-61cfc563811d-kube-api-access-g54c8\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:14.508468 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:14.508435 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-62pxm" Nov 03 00:11:14.660035 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:14.659999 6400 generic.go:356] "Generic (PLEG): container finished" podID="18531534-0eff-444b-b11d-61cfc563811d" containerID="ffba9c3005f887ebe21e70dc84506d104c51d089aaab342c34ed7e17740b6f99" exitCode=0 Nov 03 00:11:14.660523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:14.660059 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" Nov 03 00:11:14.660523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:14.660085 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" event={"ID":"18531534-0eff-444b-b11d-61cfc563811d","Type":"ContainerDied","Data":"ffba9c3005f887ebe21e70dc84506d104c51d089aaab342c34ed7e17740b6f99"} Nov 03 00:11:14.660523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:14.660137 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq" event={"ID":"18531534-0eff-444b-b11d-61cfc563811d","Type":"ContainerDied","Data":"53df5de2b5b217a71377650de1fe8cef38a580b6894f6afb9b2eb40a1b415091"} Nov 03 00:11:14.660523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:14.660159 6400 scope.go:122] "RemoveContainer" containerID="ffba9c3005f887ebe21e70dc84506d104c51d089aaab342c34ed7e17740b6f99" Nov 03 00:11:14.669531 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:14.669510 6400 scope.go:122] "RemoveContainer" containerID="ffba9c3005f887ebe21e70dc84506d104c51d089aaab342c34ed7e17740b6f99" Nov 03 00:11:14.669964 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:11:14.669938 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffba9c3005f887ebe21e70dc84506d104c51d089aaab342c34ed7e17740b6f99\": container with ID starting with ffba9c3005f887ebe21e70dc84506d104c51d089aaab342c34ed7e17740b6f99 not found: ID does not exist" containerID="ffba9c3005f887ebe21e70dc84506d104c51d089aaab342c34ed7e17740b6f99" Nov 03 00:11:14.670064 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:14.669971 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffba9c3005f887ebe21e70dc84506d104c51d089aaab342c34ed7e17740b6f99"} err="failed to get container status \"ffba9c3005f887ebe21e70dc84506d104c51d089aaab342c34ed7e17740b6f99\": rpc error: code = NotFound desc = could not find container \"ffba9c3005f887ebe21e70dc84506d104c51d089aaab342c34ed7e17740b6f99\": container with ID starting with ffba9c3005f887ebe21e70dc84506d104c51d089aaab342c34ed7e17740b6f99 not found: ID does not exist" Nov 03 00:11:14.823119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:14.823087 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq"] Nov 03 00:11:14.831844 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:14.831775 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-64d6cbf9dd-gg6nq"] Nov 03 00:11:15.178350 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:15.178273 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18531534-0eff-444b-b11d-61cfc563811d" path="/var/lib/kubelet/pods/18531534-0eff-444b-b11d-61cfc563811d/volumes" Nov 03 00:11:15.510811 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:15.510737 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" Nov 03 00:11:27.151849 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:11:27.151796 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:11:47.993335 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:47.993301 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-bg4db"] Nov 03 00:11:47.993856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:47.993692 6400 cpu_manager.go:414] "containerMap: removing container" podUID="9cb77863-cd9c-418b-91e3-96ceec1fd8fc" containerName="authorino" Nov 03 00:11:47.993856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:47.993704 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb77863-cd9c-418b-91e3-96ceec1fd8fc" containerName="authorino" Nov 03 00:11:47.993856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:47.993717 6400 cpu_manager.go:414] "containerMap: removing container" podUID="702a0068-f89c-49b7-a675-022607ad46b9" containerName="authorino" Nov 03 00:11:47.993856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:47.993722 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702a0068-f89c-49b7-a675-022607ad46b9" containerName="authorino" Nov 03 00:11:47.993856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:47.993747 6400 cpu_manager.go:414] "containerMap: removing container" podUID="18531534-0eff-444b-b11d-61cfc563811d" containerName="maas-api" Nov 03 00:11:47.993856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:47.993754 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18531534-0eff-444b-b11d-61cfc563811d" containerName="maas-api" Nov 03 00:11:47.993856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:47.993815 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="18531534-0eff-444b-b11d-61cfc563811d" containerName="maas-api" Nov 03 00:11:47.993856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:47.993838 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb77863-cd9c-418b-91e3-96ceec1fd8fc" containerName="authorino" Nov 03 00:11:47.993856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:47.993844 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="702a0068-f89c-49b7-a675-022607ad46b9" containerName="authorino" Nov 03 00:11:47.997109 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:47.997094 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bg4db" Nov 03 00:11:48.052547 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:48.052520 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-bg4db"] Nov 03 00:11:48.057972 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:48.057935 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfsr\" (UniqueName: \"kubernetes.io/projected/9adacad7-56f7-41dc-90c2-0dbdab888504-kube-api-access-6xfsr\") pod \"llmisvc-controller-manager-6466c554d8-bg4db\" (UID: \"9adacad7-56f7-41dc-90c2-0dbdab888504\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bg4db" Nov 03 00:11:48.058115 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:48.058003 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9adacad7-56f7-41dc-90c2-0dbdab888504-cert\") pod \"llmisvc-controller-manager-6466c554d8-bg4db\" (UID: \"9adacad7-56f7-41dc-90c2-0dbdab888504\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bg4db" Nov 03 00:11:48.158540 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:48.158501 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfsr\" (UniqueName: \"kubernetes.io/projected/9adacad7-56f7-41dc-90c2-0dbdab888504-kube-api-access-6xfsr\") pod \"llmisvc-controller-manager-6466c554d8-bg4db\" (UID: \"9adacad7-56f7-41dc-90c2-0dbdab888504\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bg4db" Nov 03 00:11:48.158716 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:48.158557 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9adacad7-56f7-41dc-90c2-0dbdab888504-cert\") pod \"llmisvc-controller-manager-6466c554d8-bg4db\" (UID: \"9adacad7-56f7-41dc-90c2-0dbdab888504\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bg4db" Nov 03 00:11:48.161119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:48.161094 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9adacad7-56f7-41dc-90c2-0dbdab888504-cert\") pod \"llmisvc-controller-manager-6466c554d8-bg4db\" (UID: \"9adacad7-56f7-41dc-90c2-0dbdab888504\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bg4db" Nov 03 00:11:48.215396 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:48.215364 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfsr\" (UniqueName: \"kubernetes.io/projected/9adacad7-56f7-41dc-90c2-0dbdab888504-kube-api-access-6xfsr\") pod \"llmisvc-controller-manager-6466c554d8-bg4db\" (UID: \"9adacad7-56f7-41dc-90c2-0dbdab888504\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bg4db" Nov 03 00:11:48.309319 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:48.309297 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bg4db" Nov 03 00:11:48.677258 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:48.677047 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-bg4db"] Nov 03 00:11:48.680107 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:11:48.680067 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adacad7_56f7_41dc_90c2_0dbdab888504.slice/crio-204e248e4f1b51ac1780fc74331eec8b311bb0cb454952c554287f02b11c25b3 WatchSource:0}: Error finding container 204e248e4f1b51ac1780fc74331eec8b311bb0cb454952c554287f02b11c25b3: Status 404 returned error can't find the container with id 204e248e4f1b51ac1780fc74331eec8b311bb0cb454952c554287f02b11c25b3 Nov 03 00:11:48.681401 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:48.681381 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:11:48.815233 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:48.815202 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bg4db" event={"ID":"9adacad7-56f7-41dc-90c2-0dbdab888504","Type":"ContainerStarted","Data":"204e248e4f1b51ac1780fc74331eec8b311bb0cb454952c554287f02b11c25b3"} Nov 03 00:11:49.827597 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:49.827551 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bg4db" event={"ID":"9adacad7-56f7-41dc-90c2-0dbdab888504","Type":"ContainerStarted","Data":"27650eb8e8d18df769416fd25f98581c4ddf12b5e00257fafef9d09d84edd5d9"} Nov 03 00:11:49.828049 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:49.827700 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bg4db" Nov 03 00:11:49.940725 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:11:49.940673 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bg4db" podStartSLOduration=2.125050421 podStartE2EDuration="2.940659421s" totalImagesPullingTime="815.609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:49.939475615 +0000 UTC m=+356.890415773" watchObservedRunningTime="2026-11-03 00:11:49.940659421 +0000 UTC m=+356.891599577" Nov 03 00:11:57.152417 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:11:57.152378 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:12:00.084296 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:00.084261 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:16.318887 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:16.318793 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:18.283511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:18.283471 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:20.834778 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:20.834749 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bg4db" Nov 03 00:12:20.998352 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:20.998314 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc"] Nov 03 00:12:20.998623 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:20.998596 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" podUID="0e874e2e-b7f2-48f2-b538-7632ed57ecb0" containerName="manager" containerID="cri-o://0ca6aabb0fe797058398a9b49de39d51282d36118b94012efaf427745bfeb6b0" gracePeriod=30 Nov 03 00:12:21.248532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.248505 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" Nov 03 00:12:21.362789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.362487 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-kube-api-access-r5p2b\" (UniqueName: \"kubernetes.io/projected/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-kube-api-access-r5p2b\") pod \"0e874e2e-b7f2-48f2-b538-7632ed57ecb0\" (UID: \"0e874e2e-b7f2-48f2-b538-7632ed57ecb0\") " Nov 03 00:12:21.362789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.362601 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-cert\" (UniqueName: \"kubernetes.io/secret/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-cert\") pod \"0e874e2e-b7f2-48f2-b538-7632ed57ecb0\" (UID: \"0e874e2e-b7f2-48f2-b538-7632ed57ecb0\") " Nov 03 00:12:21.365287 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.365251 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-cert" pod "0e874e2e-b7f2-48f2-b538-7632ed57ecb0" (UID: "0e874e2e-b7f2-48f2-b538-7632ed57ecb0"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:12:21.365433 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.365357 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-kube-api-access-r5p2b" pod "0e874e2e-b7f2-48f2-b538-7632ed57ecb0" (UID: "0e874e2e-b7f2-48f2-b538-7632ed57ecb0"). InnerVolumeSpecName "kube-api-access-r5p2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:12:21.463966 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.463927 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5p2b\" (UniqueName: \"kubernetes.io/projected/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-kube-api-access-r5p2b\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:21.463966 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.463959 6400 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e874e2e-b7f2-48f2-b538-7632ed57ecb0-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:21.972262 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.972226 6400 generic.go:356] "Generic (PLEG): container finished" podID="0e874e2e-b7f2-48f2-b538-7632ed57ecb0" containerID="0ca6aabb0fe797058398a9b49de39d51282d36118b94012efaf427745bfeb6b0" exitCode=0 Nov 03 00:12:21.972778 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.972278 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" Nov 03 00:12:21.972778 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.972315 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" event={"ID":"0e874e2e-b7f2-48f2-b538-7632ed57ecb0","Type":"ContainerDied","Data":"0ca6aabb0fe797058398a9b49de39d51282d36118b94012efaf427745bfeb6b0"} Nov 03 00:12:21.972778 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.972384 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc" event={"ID":"0e874e2e-b7f2-48f2-b538-7632ed57ecb0","Type":"ContainerDied","Data":"b7a3f4093ab543773489498e3f27327e9f2604c58a57496ab1cf6872cc24349a"} Nov 03 00:12:21.972778 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.972419 6400 scope.go:122] "RemoveContainer" containerID="0ca6aabb0fe797058398a9b49de39d51282d36118b94012efaf427745bfeb6b0" Nov 03 00:12:21.981488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.981470 6400 scope.go:122] "RemoveContainer" containerID="0ca6aabb0fe797058398a9b49de39d51282d36118b94012efaf427745bfeb6b0" Nov 03 00:12:21.981761 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:12:21.981741 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca6aabb0fe797058398a9b49de39d51282d36118b94012efaf427745bfeb6b0\": container with ID starting with 0ca6aabb0fe797058398a9b49de39d51282d36118b94012efaf427745bfeb6b0 not found: ID does not exist" containerID="0ca6aabb0fe797058398a9b49de39d51282d36118b94012efaf427745bfeb6b0" Nov 03 00:12:21.981877 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:21.981772 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca6aabb0fe797058398a9b49de39d51282d36118b94012efaf427745bfeb6b0"} err="failed to get container status \"0ca6aabb0fe797058398a9b49de39d51282d36118b94012efaf427745bfeb6b0\": rpc error: code = NotFound desc = could not find container \"0ca6aabb0fe797058398a9b49de39d51282d36118b94012efaf427745bfeb6b0\": container with ID starting with 0ca6aabb0fe797058398a9b49de39d51282d36118b94012efaf427745bfeb6b0 not found: ID does not exist" Nov 03 00:12:22.135497 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:22.135461 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc"] Nov 03 00:12:22.138750 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:22.138724 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sw2dc"] Nov 03 00:12:22.147438 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:22.147409 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:22.374770 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:22.374738 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:23.178015 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:23.177977 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e874e2e-b7f2-48f2-b538-7632ed57ecb0" path="/var/lib/kubelet/pods/0e874e2e-b7f2-48f2-b538-7632ed57ecb0/volumes" Nov 03 00:12:27.020044 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:27.020009 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:27.153058 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:12:27.153008 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:12:29.659200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:29.659166 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:29.860086 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:29.860056 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:53.128658 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:53.128624 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:53.625225 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:53.625184 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:54.796899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:54.796865 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:55.034260 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:12:55.034216 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:57.152604 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:12:57.152569 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:13:00.469280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:00.469242 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:01.448404 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:01.448365 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:03.325280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:03.325244 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:05.244226 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:05.244169 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:13:07.834624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:07.834590 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:07.988989 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:07.988955 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:11.935041 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:11.935002 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:12.098723 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:12.098687 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:15.693018 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:15.692974 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:16.155573 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:16.155536 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:13:19.285998 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:19.285958 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:22.644130 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:22.644086 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:22.819033 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:22.818991 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:22.977192 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:22.977102 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:26.528721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:26.528668 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:27.153220 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:13:27.153186 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:13:29.431968 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:29.431921 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:13:30.297132 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:30.297093 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:42.240388 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:42.240350 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:13:49.368610 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.368573 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk"] Nov 03 00:13:49.369056 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.369037 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0e874e2e-b7f2-48f2-b538-7632ed57ecb0" containerName="manager" Nov 03 00:13:49.369056 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.369051 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e874e2e-b7f2-48f2-b538-7632ed57ecb0" containerName="manager" Nov 03 00:13:49.369145 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.369134 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e874e2e-b7f2-48f2-b538-7632ed57ecb0" containerName="manager" Nov 03 00:13:49.372134 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.372111 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.456461 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.456427 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zx4cj\"" Nov 03 00:13:49.475202 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.475173 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk"] Nov 03 00:13:49.496330 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.496302 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.496500 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.496348 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956c60bc-c036-450b-85d9-ffd5f5a0971e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.496500 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.496375 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.496500 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.496480 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956c60bc-c036-450b-85d9-ffd5f5a0971e-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.496658 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.496519 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj8dm\" (UniqueName: \"kubernetes.io/projected/956c60bc-c036-450b-85d9-ffd5f5a0971e-kube-api-access-mj8dm\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.496658 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.496543 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.496658 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.496581 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.496810 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.496673 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956c60bc-c036-450b-85d9-ffd5f5a0971e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.496810 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.496693 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.597551 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.597515 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.597551 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.597558 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.597789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.597682 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956c60bc-c036-450b-85d9-ffd5f5a0971e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.597789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.597723 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.597789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.597748 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956c60bc-c036-450b-85d9-ffd5f5a0971e-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.597978 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.597792 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj8dm\" (UniqueName: \"kubernetes.io/projected/956c60bc-c036-450b-85d9-ffd5f5a0971e-kube-api-access-mj8dm\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.597978 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.597855 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.597978 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.597916 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.597978 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.597958 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.598175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.597982 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956c60bc-c036-450b-85d9-ffd5f5a0971e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.598175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.598108 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.598422 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.598400 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.598488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.598451 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.598645 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.598628 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956c60bc-c036-450b-85d9-ffd5f5a0971e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.600492 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.600469 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956c60bc-c036-450b-85d9-ffd5f5a0971e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.600594 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.600530 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956c60bc-c036-450b-85d9-ffd5f5a0971e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.655840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.655757 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj8dm\" (UniqueName: \"kubernetes.io/projected/956c60bc-c036-450b-85d9-ffd5f5a0971e-kube-api-access-mj8dm\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.655840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.655760 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956c60bc-c036-450b-85d9-ffd5f5a0971e-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk\" (UID: \"956c60bc-c036-450b-85d9-ffd5f5a0971e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:49.685774 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:49.685742 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:50.115904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:50.115868 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk"] Nov 03 00:13:50.156545 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:13:50.156509 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956c60bc_c036_450b_85d9_ffd5f5a0971e.slice/crio-5ad53194efe0919ea47922728f3236092ace018e50c1b94999e550cc3cfb5987 WatchSource:0}: Error finding container 5ad53194efe0919ea47922728f3236092ace018e50c1b94999e550cc3cfb5987: Status 404 returned error can't find the container with id 5ad53194efe0919ea47922728f3236092ace018e50c1b94999e550cc3cfb5987 Nov 03 00:13:50.381046 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:50.380944 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" event={"ID":"956c60bc-c036-450b-85d9-ffd5f5a0971e","Type":"ContainerStarted","Data":"5ad53194efe0919ea47922728f3236092ace018e50c1b94999e550cc3cfb5987"} Nov 03 00:13:54.516426 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:54.516388 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:13:54.516722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:54.516463 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:13:54.516722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:54.516497 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:13:55.405983 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:55.405951 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" event={"ID":"956c60bc-c036-450b-85d9-ffd5f5a0971e","Type":"ContainerStarted","Data":"468207b753e59092ed9b2db448c7922e21bbd66947ac775d74f3386efbe63d35"} Nov 03 00:13:55.596848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:55.591355 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" podStartSLOduration=2.234353349 podStartE2EDuration="6.591333376s" totalImagesPullingTime="4.356980027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:55.570905993 +0000 UTC m=+482.521846159" watchObservedRunningTime="2026-11-03 00:13:55.591333376 +0000 UTC m=+482.542273534" Nov 03 00:13:55.697905 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:55.697506 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:13:55.697905 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:55.697683 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:13:55.697905 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:55.697745 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" podUID="956c60bc-c036-450b-85d9-ffd5f5a0971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:13:56.694847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:56.694233 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:13:56.694847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:56.694315 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" podUID="956c60bc-c036-450b-85d9-ffd5f5a0971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:13:57.153209 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:13:57.153173 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:13:57.701023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:57.700976 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:13:57.701556 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:57.701075 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" podUID="956c60bc-c036-450b-85d9-ffd5f5a0971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:13:58.160954 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:58.160915 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:58.698849 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:58.698378 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:13:58.698849 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:58.698460 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" podUID="956c60bc-c036-450b-85d9-ffd5f5a0971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:13:59.347988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:59.347945 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:59.649593 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:59.649501 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:59.692349 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:59.692312 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:13:59.692635 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:59.692601 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" podUID="956c60bc-c036-450b-85d9-ffd5f5a0971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:13:59.820797 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:13:59.820754 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:00.045146 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.045110 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69cd468fff-k2b5p"] Nov 03 00:14:00.045313 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.045292 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-69cd468fff-k2b5p" podUID="421a7e2c-a20f-45f8-a213-e338caf92496" containerName="authorino" containerID="cri-o://0808dce92ecbcae6a0ee76b3d770e3745c5423cc49bdbf4acaaaa65b4ec2b8e8" gracePeriod=30 Nov 03 00:14:00.348360 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.348333 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69cd468fff-k2b5p" Nov 03 00:14:00.420716 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.420684 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421a7e2c-a20f-45f8-a213-e338caf92496-kube-api-access-l6tj2\" (UniqueName: \"kubernetes.io/projected/421a7e2c-a20f-45f8-a213-e338caf92496-kube-api-access-l6tj2\") pod \"421a7e2c-a20f-45f8-a213-e338caf92496\" (UID: \"421a7e2c-a20f-45f8-a213-e338caf92496\") " Nov 03 00:14:00.420920 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.420753 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/421a7e2c-a20f-45f8-a213-e338caf92496-tls-cert\" (UniqueName: \"kubernetes.io/secret/421a7e2c-a20f-45f8-a213-e338caf92496-tls-cert\") pod \"421a7e2c-a20f-45f8-a213-e338caf92496\" (UID: \"421a7e2c-a20f-45f8-a213-e338caf92496\") " Nov 03 00:14:00.423185 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.423154 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421a7e2c-a20f-45f8-a213-e338caf92496-kube-api-access-l6tj2" pod "421a7e2c-a20f-45f8-a213-e338caf92496" (UID: "421a7e2c-a20f-45f8-a213-e338caf92496"). InnerVolumeSpecName "kube-api-access-l6tj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:00.432381 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.432343 6400 generic.go:356] "Generic (PLEG): container finished" podID="421a7e2c-a20f-45f8-a213-e338caf92496" containerID="0808dce92ecbcae6a0ee76b3d770e3745c5423cc49bdbf4acaaaa65b4ec2b8e8" exitCode=0 Nov 03 00:14:00.432523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.432390 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69cd468fff-k2b5p" Nov 03 00:14:00.432523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.432425 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69cd468fff-k2b5p" event={"ID":"421a7e2c-a20f-45f8-a213-e338caf92496","Type":"ContainerDied","Data":"0808dce92ecbcae6a0ee76b3d770e3745c5423cc49bdbf4acaaaa65b4ec2b8e8"} Nov 03 00:14:00.432523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.432471 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69cd468fff-k2b5p" event={"ID":"421a7e2c-a20f-45f8-a213-e338caf92496","Type":"ContainerDied","Data":"dfcf85006a4ef56ad3d37b3873c8bb3b2aeffdc0b0aafad528d9fdce4ef39550"} Nov 03 00:14:00.432523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.432488 6400 scope.go:122] "RemoveContainer" containerID="0808dce92ecbcae6a0ee76b3d770e3745c5423cc49bdbf4acaaaa65b4ec2b8e8" Nov 03 00:14:00.434754 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.434725 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/421a7e2c-a20f-45f8-a213-e338caf92496-tls-cert" pod "421a7e2c-a20f-45f8-a213-e338caf92496" (UID: "421a7e2c-a20f-45f8-a213-e338caf92496"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:14:00.443751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.443738 6400 scope.go:122] "RemoveContainer" containerID="0808dce92ecbcae6a0ee76b3d770e3745c5423cc49bdbf4acaaaa65b4ec2b8e8" Nov 03 00:14:00.444048 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:14:00.444030 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0808dce92ecbcae6a0ee76b3d770e3745c5423cc49bdbf4acaaaa65b4ec2b8e8\": container with ID starting with 0808dce92ecbcae6a0ee76b3d770e3745c5423cc49bdbf4acaaaa65b4ec2b8e8 not found: ID does not exist" containerID="0808dce92ecbcae6a0ee76b3d770e3745c5423cc49bdbf4acaaaa65b4ec2b8e8" Nov 03 00:14:00.444110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.444059 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0808dce92ecbcae6a0ee76b3d770e3745c5423cc49bdbf4acaaaa65b4ec2b8e8"} err="failed to get container status \"0808dce92ecbcae6a0ee76b3d770e3745c5423cc49bdbf4acaaaa65b4ec2b8e8\": rpc error: code = NotFound desc = could not find container \"0808dce92ecbcae6a0ee76b3d770e3745c5423cc49bdbf4acaaaa65b4ec2b8e8\": container with ID starting with 0808dce92ecbcae6a0ee76b3d770e3745c5423cc49bdbf4acaaaa65b4ec2b8e8 not found: ID does not exist" Nov 03 00:14:00.521941 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.521905 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6tj2\" (UniqueName: \"kubernetes.io/projected/421a7e2c-a20f-45f8-a213-e338caf92496-kube-api-access-l6tj2\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:00.521941 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.521934 6400 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/421a7e2c-a20f-45f8-a213-e338caf92496-tls-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:00.686724 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.686637 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:14:00.686906 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.686725 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" podUID="956c60bc-c036-450b-85d9-ffd5f5a0971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:14:00.778286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.778251 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:00.854676 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.854643 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69cd468fff-k2b5p"] Nov 03 00:14:00.858518 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:00.858491 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-69cd468fff-k2b5p"] Nov 03 00:14:01.178058 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:01.178027 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421a7e2c-a20f-45f8-a213-e338caf92496" path="/var/lib/kubelet/pods/421a7e2c-a20f-45f8-a213-e338caf92496/volumes" Nov 03 00:14:01.686774 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:01.686735 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 00:14:01.687208 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:01.686805 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" podUID="956c60bc-c036-450b-85d9-ffd5f5a0971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 00:14:02.837663 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:02.837630 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:14:02.838480 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:02.838457 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:14:02.839203 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:02.839153 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" Nov 03 00:14:27.151676 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:14:27.151631 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:14:35.523432 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:35.523385 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:35.756598 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:35.756561 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:36.049175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:36.049133 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:36.303936 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:36.303801 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:36.487260 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:36.487222 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:36.653146 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:36.653055 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:36.893669 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:36.893626 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:37.237759 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:37.237721 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:37.391046 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:37.391008 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:38.684658 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:38.684620 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:38.887924 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:38.887882 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:39.305389 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:39.305350 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:39.487643 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:39.487603 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:39.679005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:39.678911 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:40.262630 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:40.262592 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:40.474700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:40.474656 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:42.059602 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:42.059562 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:42.226375 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:42.226341 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:42.385768 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:42.385671 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:42.702250 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:42.702151 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:42.914195 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:42.914153 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:43.080458 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:43.080421 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:43.248701 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:43.248661 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:44.270319 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:44.270276 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:44.478582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:44.478547 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:44.649724 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:44.649687 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:45.584594 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:45.584558 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:45.795181 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:45.795137 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:45.965441 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:45.965355 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:47.563835 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:47.563774 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:47.758805 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:47.758772 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:50.051219 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:50.051169 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:50.229421 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:50.229379 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:50.404503 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:50.404403 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:50.599657 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:50.599618 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:50.757574 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:50.757484 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:50.927978 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:50.927937 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:51.098207 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:51.098165 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:52.695509 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:52.695469 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:52.863925 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:52.863890 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:54.755641 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:54.755610 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:14:55.737268 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.737226 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v"] Nov 03 00:14:55.737646 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.737633 6400 cpu_manager.go:414] "containerMap: removing container" podUID="421a7e2c-a20f-45f8-a213-e338caf92496" containerName="authorino" Nov 03 00:14:55.737692 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.737647 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421a7e2c-a20f-45f8-a213-e338caf92496" containerName="authorino" Nov 03 00:14:55.737736 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.737710 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="421a7e2c-a20f-45f8-a213-e338caf92496" containerName="authorino" Nov 03 00:14:55.741990 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.741968 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:55.795787 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.795755 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:14:55.845175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.845146 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2ccce380-serving-cert\"" Nov 03 00:14:55.845369 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.845286 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kgpd5\"" Nov 03 00:14:55.845369 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.845303 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:14:55.845369 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.845293 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:14:55.846274 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.846256 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v"] Nov 03 00:14:55.846687 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.846673 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-metrics-certs\") pod \"maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:55.846728 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.846703 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/366d22f0-45f2-4819-9a12-cc96993fb956-openshift-service-ca\") pod \"maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:55.846777 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.846729 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-maas-api-tls\") pod \"maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:55.846777 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.846747 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpgv5\" (UniqueName: \"kubernetes.io/projected/366d22f0-45f2-4819-9a12-cc96993fb956-kube-api-access-vpgv5\") pod \"maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:55.947585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.947548 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-metrics-certs\") pod \"maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:55.947807 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.947596 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/366d22f0-45f2-4819-9a12-cc96993fb956-openshift-service-ca\") pod \"maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:55.947807 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.947625 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-maas-api-tls\") pod \"maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:55.947807 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:14:55.947711 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-serving-cert: secret "maas-api-e2e-ait-2ccce380-serving-cert" not found Nov 03 00:14:55.947807 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.947747 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpgv5\" (UniqueName: \"kubernetes.io/projected/366d22f0-45f2-4819-9a12-cc96993fb956-kube-api-access-vpgv5\") pod \"maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:55.947807 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:14:55.947781 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-maas-api-tls podName:366d22f0-45f2-4819-9a12-cc96993fb956 nodeName:}" failed. No retries permitted until 2026-11-03 00:14:56.447761299 +0000 UTC m=+543.398701448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-maas-api-tls") pod "maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" (UID: "366d22f0-45f2-4819-9a12-cc96993fb956") : secret "maas-api-e2e-ait-2ccce380-serving-cert" not found Nov 03 00:14:55.948283 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.948260 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/366d22f0-45f2-4819-9a12-cc96993fb956-openshift-service-ca\") pod \"maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:55.950380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.950357 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-metrics-certs\") pod \"maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:55.956034 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:55.956008 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:56.005156 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:56.005092 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpgv5\" (UniqueName: \"kubernetes.io/projected/366d22f0-45f2-4819-9a12-cc96993fb956-kube-api-access-vpgv5\") pod \"maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:56.157666 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:56.157617 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:56.320552 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:56.320521 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:56.456065 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:56.456025 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-maas-api-tls\") pod \"maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:56.458597 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:56.458569 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-maas-api-tls\") pod \"maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:56.654248 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:56.654160 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:57.031131 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:14:57.031097 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366d22f0_45f2_4819_9a12_cc96993fb956.slice/crio-746ab48c3cb5fc6511016b9025997279aec9b93bde7dd20fc56c91b0d21ffda3 WatchSource:0}: Error finding container 746ab48c3cb5fc6511016b9025997279aec9b93bde7dd20fc56c91b0d21ffda3: Status 404 returned error can't find the container with id 746ab48c3cb5fc6511016b9025997279aec9b93bde7dd20fc56c91b0d21ffda3 Nov 03 00:14:57.056543 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:57.056512 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v"] Nov 03 00:14:57.151673 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:14:57.151640 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:14:57.715078 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:57.715039 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" event={"ID":"366d22f0-45f2-4819-9a12-cc96993fb956","Type":"ContainerStarted","Data":"746ab48c3cb5fc6511016b9025997279aec9b93bde7dd20fc56c91b0d21ffda3"} Nov 03 00:14:58.721497 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:58.721457 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" event={"ID":"366d22f0-45f2-4819-9a12-cc96993fb956","Type":"ContainerStarted","Data":"dbb9c0d69e70680e6b38a66ea1cb6dc501260ff20575be97dcc875e4974e3268"} Nov 03 00:14:58.722021 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:58.721524 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:14:58.838168 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:14:58.838114 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" podStartSLOduration=2.937641723 podStartE2EDuration="3.838098822s" totalImagesPullingTime="900.457099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:58.83717918 +0000 UTC m=+545.788119341" watchObservedRunningTime="2026-11-03 00:14:58.838098822 +0000 UTC m=+545.789038979" Nov 03 00:15:00.162728 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:00.162690 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4"] Nov 03 00:15:00.166403 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:00.166385 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" Nov 03 00:15:00.265947 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:00.265908 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4"] Nov 03 00:15:00.292595 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:00.292559 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4\" (UID: \"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" Nov 03 00:15:00.292799 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:00.292608 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlrx6\" (UniqueName: \"kubernetes.io/projected/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-kube-api-access-mlrx6\") pod \"maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4\" (UID: \"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" Nov 03 00:15:00.313931 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:00.313895 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:15:00.393469 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:00.393436 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4\" (UID: \"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" Nov 03 00:15:00.393649 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:00.393486 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlrx6\" (UniqueName: \"kubernetes.io/projected/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-kube-api-access-mlrx6\") pod \"maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4\" (UID: \"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" Nov 03 00:15:00.394103 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:00.394084 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4\" (UID: \"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" Nov 03 00:15:00.450194 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:00.450121 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlrx6\" (UniqueName: \"kubernetes.io/projected/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-kube-api-access-mlrx6\") pod \"maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4\" (UID: \"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" Nov 03 00:15:00.478655 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:00.478627 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" Nov 03 00:15:00.849636 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:00.849601 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4"] Nov 03 00:15:00.852702 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:15:00.852674 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6d91d7_0b24_4b9e_954a_fe3b5e264b4e.slice/crio-eb006a1f3fb3651d188c5e5c2c1e4bd659872945cd8c6759c3e8814a4fbc94d3 WatchSource:0}: Error finding container eb006a1f3fb3651d188c5e5c2c1e4bd659872945cd8c6759c3e8814a4fbc94d3: Status 404 returned error can't find the container with id eb006a1f3fb3651d188c5e5c2c1e4bd659872945cd8c6759c3e8814a4fbc94d3 Nov 03 00:15:01.029524 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:01.029486 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:01.197985 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:01.197899 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:01.740598 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:01.740563 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" event={"ID":"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e","Type":"ContainerStarted","Data":"eb006a1f3fb3651d188c5e5c2c1e4bd659872945cd8c6759c3e8814a4fbc94d3"} Nov 03 00:15:02.357317 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:02.357289 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:02.612225 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:02.612120 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:04.020982 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:04.020951 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:15:04.731437 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:04.731406 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:15:05.763410 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:05.763327 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" event={"ID":"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e","Type":"ContainerStarted","Data":"d22bd444fde752b9580cdcccf837aa8afca9344fd3d864c2f894af47de18c95e"} Nov 03 00:15:05.878604 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:05.878550 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" podStartSLOduration=1.5893570989999999 podStartE2EDuration="5.878536493s" totalImagesPullingTime="4.289179394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:05.877678999 +0000 UTC m=+552.828619158" watchObservedRunningTime="2026-11-03 00:15:05.878536493 +0000 UTC m=+552.829476650" Nov 03 00:15:05.918681 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:05.918633 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:06.127563 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:06.127529 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:06.433318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:06.433216 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:06.769637 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:06.769547 6400 generic.go:356] "Generic (PLEG): container finished" podID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" containerID="d22bd444fde752b9580cdcccf837aa8afca9344fd3d864c2f894af47de18c95e" exitCode=60 Nov 03 00:15:06.770044 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:06.769640 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" event={"ID":"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e","Type":"ContainerDied","Data":"d22bd444fde752b9580cdcccf837aa8afca9344fd3d864c2f894af47de18c95e"} Nov 03 00:15:06.770044 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:06.769869 6400 scope.go:122] "RemoveContainer" containerID="d22bd444fde752b9580cdcccf837aa8afca9344fd3d864c2f894af47de18c95e" Nov 03 00:15:07.784215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:07.784166 6400 generic.go:356] "Generic (PLEG): container finished" podID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" containerID="7b5d2c3c8c3c753ef5889cb4fd395eca235cff039992d6240acbfcc51ff16860" exitCode=60 Nov 03 00:15:07.784642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:07.784239 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" event={"ID":"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e","Type":"ContainerDied","Data":"7b5d2c3c8c3c753ef5889cb4fd395eca235cff039992d6240acbfcc51ff16860"} Nov 03 00:15:07.784642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:07.784296 6400 scope.go:122] "RemoveContainer" containerID="d22bd444fde752b9580cdcccf837aa8afca9344fd3d864c2f894af47de18c95e" Nov 03 00:15:07.784642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:07.784585 6400 scope.go:122] "RemoveContainer" containerID="7b5d2c3c8c3c753ef5889cb4fd395eca235cff039992d6240acbfcc51ff16860" Nov 03 00:15:07.784895 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:15:07.784868 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4_odh-ai-gateway-infra(ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" podUID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" Nov 03 00:15:08.791469 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:08.791442 6400 scope.go:122] "RemoveContainer" containerID="7b5d2c3c8c3c753ef5889cb4fd395eca235cff039992d6240acbfcc51ff16860" Nov 03 00:15:08.791958 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:15:08.791723 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4_odh-ai-gateway-infra(ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" podUID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" Nov 03 00:15:10.526989 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.526952 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz"] Nov 03 00:15:10.532130 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.532107 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:10.586154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.586126 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c6f27688-serving-cert\"" Nov 03 00:15:10.636969 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.636934 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz"] Nov 03 00:15:10.694215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.694183 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-maas-api-tls\") pod \"maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:10.694215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.694215 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjrrv\" (UniqueName: \"kubernetes.io/projected/84a4fdce-06fa-465d-95a3-092ba679cdea-kube-api-access-hjrrv\") pod \"maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:10.694416 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.694300 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-metrics-certs\") pod \"maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:10.694416 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.694400 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a4fdce-06fa-465d-95a3-092ba679cdea-openshift-service-ca\") pod \"maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:10.749897 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.749861 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:10.795693 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.795657 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a4fdce-06fa-465d-95a3-092ba679cdea-openshift-service-ca\") pod \"maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:10.795918 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.795748 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-maas-api-tls\") pod \"maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:10.795918 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.795766 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjrrv\" (UniqueName: \"kubernetes.io/projected/84a4fdce-06fa-465d-95a3-092ba679cdea-kube-api-access-hjrrv\") pod \"maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:10.795918 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.795794 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-metrics-certs\") pod \"maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:10.795918 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:15:10.795905 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-serving-cert: secret "maas-api-e2e-migrate-c6f27688-serving-cert" not found Nov 03 00:15:10.796122 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:15:10.795985 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-maas-api-tls podName:84a4fdce-06fa-465d-95a3-092ba679cdea nodeName:}" failed. No retries permitted until 2026-11-03 00:15:11.295963588 +0000 UTC m=+558.246903741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-maas-api-tls") pod "maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" (UID: "84a4fdce-06fa-465d-95a3-092ba679cdea") : secret "maas-api-e2e-migrate-c6f27688-serving-cert" not found Nov 03 00:15:10.796521 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.796493 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a4fdce-06fa-465d-95a3-092ba679cdea-openshift-service-ca\") pod \"maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:10.798764 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.798742 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-metrics-certs\") pod \"maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:10.854732 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.854701 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjrrv\" (UniqueName: \"kubernetes.io/projected/84a4fdce-06fa-465d-95a3-092ba679cdea-kube-api-access-hjrrv\") pod \"maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:10.966137 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:10.966099 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:11.135656 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:11.135565 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:11.301176 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:11.301146 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-maas-api-tls\") pod \"maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:11.303695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:11.303664 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-maas-api-tls\") pod \"maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:11.445023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:11.444925 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:11.819992 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:11.819966 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz"] Nov 03 00:15:11.822738 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:15:11.822703 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a4fdce_06fa_465d_95a3_092ba679cdea.slice/crio-be87b6bc92642dd2f35395a40aeb81770a974700d8fc6b073389d7d145bae642 WatchSource:0}: Error finding container be87b6bc92642dd2f35395a40aeb81770a974700d8fc6b073389d7d145bae642: Status 404 returned error can't find the container with id be87b6bc92642dd2f35395a40aeb81770a974700d8fc6b073389d7d145bae642 Nov 03 00:15:12.811053 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:12.811007 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" event={"ID":"84a4fdce-06fa-465d-95a3-092ba679cdea","Type":"ContainerStarted","Data":"24973a3b975e7b822c3356ea7df153ca0a299392d3a650551cec0cb5395807bb"} Nov 03 00:15:12.811243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:12.811153 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:12.811243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:12.811175 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" event={"ID":"84a4fdce-06fa-465d-95a3-092ba679cdea","Type":"ContainerStarted","Data":"be87b6bc92642dd2f35395a40aeb81770a974700d8fc6b073389d7d145bae642"} Nov 03 00:15:12.927175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:12.927072 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" podStartSLOduration=2.099754338 podStartE2EDuration="2.927054019s" totalImagesPullingTime="827.299681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:12.92660478 +0000 UTC m=+559.877544940" watchObservedRunningTime="2026-11-03 00:15:12.927054019 +0000 UTC m=+559.877994180" Nov 03 00:15:15.424476 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:15.424432 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:15.467934 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:15.467897 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:15:18.820748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:18.820714 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:15:20.196602 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:20.196551 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:21.172889 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:21.172855 6400 scope.go:122] "RemoveContainer" containerID="7b5d2c3c8c3c753ef5889cb4fd395eca235cff039992d6240acbfcc51ff16860" Nov 03 00:15:21.860570 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:21.860532 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" event={"ID":"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e","Type":"ContainerStarted","Data":"042ea1214a7f8f95e6312d6f73808474fdf639ec63fa9c89cf419359f9ae184f"} Nov 03 00:15:22.867408 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:22.867374 6400 generic.go:356] "Generic (PLEG): container finished" podID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" containerID="042ea1214a7f8f95e6312d6f73808474fdf639ec63fa9c89cf419359f9ae184f" exitCode=60 Nov 03 00:15:22.867894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:22.867422 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" event={"ID":"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e","Type":"ContainerDied","Data":"042ea1214a7f8f95e6312d6f73808474fdf639ec63fa9c89cf419359f9ae184f"} Nov 03 00:15:22.867894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:22.867472 6400 scope.go:122] "RemoveContainer" containerID="7b5d2c3c8c3c753ef5889cb4fd395eca235cff039992d6240acbfcc51ff16860" Nov 03 00:15:22.867894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:22.867882 6400 scope.go:122] "RemoveContainer" containerID="042ea1214a7f8f95e6312d6f73808474fdf639ec63fa9c89cf419359f9ae184f" Nov 03 00:15:22.868163 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:15:22.868141 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4_odh-ai-gateway-infra(ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" podUID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" Nov 03 00:15:22.987431 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:22.987391 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4"] Nov 03 00:15:24.012113 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:24.011634 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" Nov 03 00:15:24.114090 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:24.114052 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-kube-api-access-mlrx6\" (UniqueName: \"kubernetes.io/projected/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-kube-api-access-mlrx6\") pod \"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e\" (UID: \"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e\") " Nov 03 00:15:24.114266 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:24.114106 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-maas-api-service-ca\") pod \"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e\" (UID: \"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e\") " Nov 03 00:15:24.114488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:24.114456 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-maas-api-service-ca" pod "ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" (UID: "ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:24.116339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:24.116314 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-kube-api-access-mlrx6" pod "ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" (UID: "ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e"). InnerVolumeSpecName "kube-api-access-mlrx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:24.215335 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:24.215254 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlrx6\" (UniqueName: \"kubernetes.io/projected/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-kube-api-access-mlrx6\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:24.215335 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:24.215281 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:24.262582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:24.262522 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:15:24.878714 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:24.878686 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" Nov 03 00:15:24.878714 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:24.878693 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4" event={"ID":"ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e","Type":"ContainerDied","Data":"eb006a1f3fb3651d188c5e5c2c1e4bd659872945cd8c6759c3e8814a4fbc94d3"} Nov 03 00:15:24.878971 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:24.878749 6400 scope.go:122] "RemoveContainer" containerID="042ea1214a7f8f95e6312d6f73808474fdf639ec63fa9c89cf419359f9ae184f" Nov 03 00:15:25.042166 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:25.042136 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4"] Nov 03 00:15:25.047965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:25.047940 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-12537192-29894415-r52b4"] Nov 03 00:15:25.177850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:25.177759 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" path="/var/lib/kubelet/pods/ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e/volumes" Nov 03 00:15:25.395442 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:25.395400 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:27.152143 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:15:27.152108 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:15:30.428198 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:30.428161 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:35.420917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:35.420885 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:38.992601 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:38.992559 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:39.213547 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:39.213507 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:40.432562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:40.432530 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:42.881562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:42.881528 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:43.096518 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:43.096481 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:44.600087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:44.600040 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:45.984121 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:45.984091 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:46.317405 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:46.317369 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:46.597392 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:46.597305 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:47.648638 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:47.648599 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:47.867053 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:47.867019 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:48.038681 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:48.038645 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:48.301304 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:48.301208 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:48.785139 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:48.785101 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:49.060320 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:49.060235 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:49.583103 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:49.583065 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:49.748751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:49.748712 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:50.034657 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:50.034615 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:50.303145 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:50.303053 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:50.493961 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:50.493920 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:51.332168 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:51.332136 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:15:53.153177 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:53.153142 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:53.712162 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:53.712126 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:54.063272 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:54.063240 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:54.348422 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:54.348333 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:54.524321 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:54.524285 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:54.712421 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:54.712331 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:54.914554 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:54.914504 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:55.102234 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:55.102201 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:56.284334 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:56.284298 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:57.152915 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:15:57.152881 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:15:57.775540 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:57.775507 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:57.977979 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:57.977942 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:58.160458 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:58.160422 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:59.581722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:59.581689 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:59.997502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:15:59.997413 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:00.221678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:00.221643 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:00.507364 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:00.507332 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:00.701370 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:00.699516 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:01.188336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:01.188307 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:16:01.343645 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:01.343604 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:01.693417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:01.693384 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:02.002653 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:02.002560 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:02.186679 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:02.186642 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:02.385106 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:02.385074 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:03.569577 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:03.569539 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v"] Nov 03 00:16:03.570095 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:03.569791 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" podUID="366d22f0-45f2-4819-9a12-cc96993fb956" containerName="maas-api" containerID="cri-o://dbb9c0d69e70680e6b38a66ea1cb6dc501260ff20575be97dcc875e4974e3268" gracePeriod=30 Nov 03 00:16:03.696224 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:03.696191 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:03.843356 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:03.843330 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:16:03.911987 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:03.911955 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-metrics-certs\" (UniqueName: \"kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-metrics-certs\") pod \"366d22f0-45f2-4819-9a12-cc96993fb956\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " Nov 03 00:16:03.911987 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:03.911992 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-maas-api-tls\") pod \"366d22f0-45f2-4819-9a12-cc96993fb956\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " Nov 03 00:16:03.912196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:03.912014 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366d22f0-45f2-4819-9a12-cc96993fb956-kube-api-access-vpgv5\" (UniqueName: \"kubernetes.io/projected/366d22f0-45f2-4819-9a12-cc96993fb956-kube-api-access-vpgv5\") pod \"366d22f0-45f2-4819-9a12-cc96993fb956\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " Nov 03 00:16:03.912196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:03.912128 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/366d22f0-45f2-4819-9a12-cc96993fb956-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/366d22f0-45f2-4819-9a12-cc96993fb956-openshift-service-ca\") pod \"366d22f0-45f2-4819-9a12-cc96993fb956\" (UID: \"366d22f0-45f2-4819-9a12-cc96993fb956\") " Nov 03 00:16:03.912507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:03.912481 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366d22f0-45f2-4819-9a12-cc96993fb956-openshift-service-ca" pod "366d22f0-45f2-4819-9a12-cc96993fb956" (UID: "366d22f0-45f2-4819-9a12-cc96993fb956"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:03.914416 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:03.914384 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-maas-api-tls" pod "366d22f0-45f2-4819-9a12-cc96993fb956" (UID: "366d22f0-45f2-4819-9a12-cc96993fb956"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:03.914544 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:03.914474 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-metrics-certs" pod "366d22f0-45f2-4819-9a12-cc96993fb956" (UID: "366d22f0-45f2-4819-9a12-cc96993fb956"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:03.914544 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:03.914498 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366d22f0-45f2-4819-9a12-cc96993fb956-kube-api-access-vpgv5" pod "366d22f0-45f2-4819-9a12-cc96993fb956" (UID: "366d22f0-45f2-4819-9a12-cc96993fb956"). InnerVolumeSpecName "kube-api-access-vpgv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:04.012869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:04.012805 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:04.012869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:04.012862 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/366d22f0-45f2-4819-9a12-cc96993fb956-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:04.012869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:04.012873 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpgv5\" (UniqueName: \"kubernetes.io/projected/366d22f0-45f2-4819-9a12-cc96993fb956-kube-api-access-vpgv5\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:04.013126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:04.012884 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/366d22f0-45f2-4819-9a12-cc96993fb956-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:04.071598 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:04.071562 6400 generic.go:356] "Generic (PLEG): container finished" podID="366d22f0-45f2-4819-9a12-cc96993fb956" containerID="dbb9c0d69e70680e6b38a66ea1cb6dc501260ff20575be97dcc875e4974e3268" exitCode=0 Nov 03 00:16:04.071798 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:04.071612 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" Nov 03 00:16:04.071798 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:04.071648 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" event={"ID":"366d22f0-45f2-4819-9a12-cc96993fb956","Type":"ContainerDied","Data":"dbb9c0d69e70680e6b38a66ea1cb6dc501260ff20575be97dcc875e4974e3268"} Nov 03 00:16:04.071798 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:04.071704 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v" event={"ID":"366d22f0-45f2-4819-9a12-cc96993fb956","Type":"ContainerDied","Data":"746ab48c3cb5fc6511016b9025997279aec9b93bde7dd20fc56c91b0d21ffda3"} Nov 03 00:16:04.071798 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:04.071735 6400 scope.go:122] "RemoveContainer" containerID="dbb9c0d69e70680e6b38a66ea1cb6dc501260ff20575be97dcc875e4974e3268" Nov 03 00:16:04.089597 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:04.089575 6400 scope.go:122] "RemoveContainer" containerID="dbb9c0d69e70680e6b38a66ea1cb6dc501260ff20575be97dcc875e4974e3268" Nov 03 00:16:04.090045 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:16:04.090020 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb9c0d69e70680e6b38a66ea1cb6dc501260ff20575be97dcc875e4974e3268\": container with ID starting with dbb9c0d69e70680e6b38a66ea1cb6dc501260ff20575be97dcc875e4974e3268 not found: ID does not exist" containerID="dbb9c0d69e70680e6b38a66ea1cb6dc501260ff20575be97dcc875e4974e3268" Nov 03 00:16:04.090171 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:04.090054 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb9c0d69e70680e6b38a66ea1cb6dc501260ff20575be97dcc875e4974e3268"} err="failed to get container status \"dbb9c0d69e70680e6b38a66ea1cb6dc501260ff20575be97dcc875e4974e3268\": rpc error: code = NotFound desc = could not find container \"dbb9c0d69e70680e6b38a66ea1cb6dc501260ff20575be97dcc875e4974e3268\": container with ID starting with dbb9c0d69e70680e6b38a66ea1cb6dc501260ff20575be97dcc875e4974e3268 not found: ID does not exist" Nov 03 00:16:04.234487 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:04.234455 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v"] Nov 03 00:16:04.240283 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:04.240253 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ccce380-59d6dc9dfd-ztz4v"] Nov 03 00:16:05.183381 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:05.183343 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="366d22f0-45f2-4819-9a12-cc96993fb956" path="/var/lib/kubelet/pods/366d22f0-45f2-4819-9a12-cc96993fb956/volumes" Nov 03 00:16:05.332020 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:05.331985 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:05.646569 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:05.646537 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:05.818417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:05.818381 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:06.010300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:06.010205 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:06.731273 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:06.731240 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:07.023001 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:07.022910 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:07.224097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:07.224064 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:07.404740 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:07.404710 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:07.780274 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:07.780189 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:10.245132 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:10.245098 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:16:10.944202 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:10.944160 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:11.339331 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:11.339298 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:11.523024 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:11.522986 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:12.813758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:12.813717 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:13.043898 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:13.043864 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:13.457963 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:13.457930 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:13.705900 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:13.705857 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:16.099811 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:16.099779 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:16.252895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:16.252860 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:16.435247 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:16.435146 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:17.242994 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:17.242961 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:16:20.310145 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:20.310108 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz"] Nov 03 00:16:20.431666 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:20.431624 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:21.158802 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.158739 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" podUID="84a4fdce-06fa-465d-95a3-092ba679cdea" containerName="maas-api" containerID="cri-o://24973a3b975e7b822c3356ea7df153ca0a299392d3a650551cec0cb5395807bb" gracePeriod=30 Nov 03 00:16:21.408518 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.408489 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:16:21.484876 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.484761 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a4fdce-06fa-465d-95a3-092ba679cdea-kube-api-access-hjrrv\" (UniqueName: \"kubernetes.io/projected/84a4fdce-06fa-465d-95a3-092ba679cdea-kube-api-access-hjrrv\") pod \"84a4fdce-06fa-465d-95a3-092ba679cdea\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " Nov 03 00:16:21.484876 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.484853 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-maas-api-tls\") pod \"84a4fdce-06fa-465d-95a3-092ba679cdea\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " Nov 03 00:16:21.485118 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.484922 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a4fdce-06fa-465d-95a3-092ba679cdea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a4fdce-06fa-465d-95a3-092ba679cdea-openshift-service-ca\") pod \"84a4fdce-06fa-465d-95a3-092ba679cdea\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " Nov 03 00:16:21.485118 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.484954 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-metrics-certs\") pod \"84a4fdce-06fa-465d-95a3-092ba679cdea\" (UID: \"84a4fdce-06fa-465d-95a3-092ba679cdea\") " Nov 03 00:16:21.485362 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.485319 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a4fdce-06fa-465d-95a3-092ba679cdea-openshift-service-ca" pod "84a4fdce-06fa-465d-95a3-092ba679cdea" (UID: "84a4fdce-06fa-465d-95a3-092ba679cdea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:21.487181 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.487158 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-metrics-certs" pod "84a4fdce-06fa-465d-95a3-092ba679cdea" (UID: "84a4fdce-06fa-465d-95a3-092ba679cdea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:21.487252 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.487205 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-maas-api-tls" pod "84a4fdce-06fa-465d-95a3-092ba679cdea" (UID: "84a4fdce-06fa-465d-95a3-092ba679cdea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:21.487286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.487254 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a4fdce-06fa-465d-95a3-092ba679cdea-kube-api-access-hjrrv" pod "84a4fdce-06fa-465d-95a3-092ba679cdea" (UID: "84a4fdce-06fa-465d-95a3-092ba679cdea"). InnerVolumeSpecName "kube-api-access-hjrrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:21.586160 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.586117 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a4fdce-06fa-465d-95a3-092ba679cdea-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:21.586160 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.586154 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:21.586160 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.586167 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjrrv\" (UniqueName: \"kubernetes.io/projected/84a4fdce-06fa-465d-95a3-092ba679cdea-kube-api-access-hjrrv\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:21.586430 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.586179 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a4fdce-06fa-465d-95a3-092ba679cdea-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:21.839348 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839316 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf"] Nov 03 00:16:21.839703 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839691 6400 cpu_manager.go:414] "containerMap: removing container" podUID="84a4fdce-06fa-465d-95a3-092ba679cdea" containerName="maas-api" Nov 03 00:16:21.839748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839705 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a4fdce-06fa-465d-95a3-092ba679cdea" containerName="maas-api" Nov 03 00:16:21.839748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839734 6400 cpu_manager.go:414] "containerMap: removing container" podUID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" containerName="cleanup" Nov 03 00:16:21.839748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839739 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" containerName="cleanup" Nov 03 00:16:21.839748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839746 6400 cpu_manager.go:414] "containerMap: removing container" podUID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" containerName="cleanup" Nov 03 00:16:21.839895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839751 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" containerName="cleanup" Nov 03 00:16:21.839895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839765 6400 cpu_manager.go:414] "containerMap: removing container" podUID="366d22f0-45f2-4819-9a12-cc96993fb956" containerName="maas-api" Nov 03 00:16:21.839895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839769 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="366d22f0-45f2-4819-9a12-cc96993fb956" containerName="maas-api" Nov 03 00:16:21.839895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839858 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" containerName="cleanup" Nov 03 00:16:21.839895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839866 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" containerName="cleanup" Nov 03 00:16:21.839895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839874 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="366d22f0-45f2-4819-9a12-cc96993fb956" containerName="maas-api" Nov 03 00:16:21.839895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839880 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" containerName="cleanup" Nov 03 00:16:21.839895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.839886 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a4fdce-06fa-465d-95a3-092ba679cdea" containerName="maas-api" Nov 03 00:16:21.843208 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.843186 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:21.895214 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.895187 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-e28e1a-serving-cert\"" Nov 03 00:16:21.901900 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.901870 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf"] Nov 03 00:16:21.990078 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.990047 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:21.990078 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.990083 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0806992d-fe7f-49d4-80d8-78ceb1bccf72-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:21.990300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.990203 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-metrics-certs\") pod \"maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:21.990300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:21.990255 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcp62\" (UniqueName: \"kubernetes.io/projected/0806992d-fe7f-49d4-80d8-78ceb1bccf72-kube-api-access-jcp62\") pod \"maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:22.056062 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.056030 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:22.091720 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.091635 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-metrics-certs\") pod \"maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:22.091720 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.091690 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcp62\" (UniqueName: \"kubernetes.io/projected/0806992d-fe7f-49d4-80d8-78ceb1bccf72-kube-api-access-jcp62\") pod \"maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:22.092000 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.091733 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:22.092000 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.091752 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0806992d-fe7f-49d4-80d8-78ceb1bccf72-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:22.092483 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.092463 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0806992d-fe7f-49d4-80d8-78ceb1bccf72-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:22.094348 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.094318 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:22.094447 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.094410 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-metrics-certs\") pod \"maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:22.148248 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.148221 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcp62\" (UniqueName: \"kubernetes.io/projected/0806992d-fe7f-49d4-80d8-78ceb1bccf72-kube-api-access-jcp62\") pod \"maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:22.156048 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.156022 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:22.164104 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.164077 6400 generic.go:356] "Generic (PLEG): container finished" podID="84a4fdce-06fa-465d-95a3-092ba679cdea" containerID="24973a3b975e7b822c3356ea7df153ca0a299392d3a650551cec0cb5395807bb" exitCode=0 Nov 03 00:16:22.164203 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.164126 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" Nov 03 00:16:22.164203 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.164140 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" event={"ID":"84a4fdce-06fa-465d-95a3-092ba679cdea","Type":"ContainerDied","Data":"24973a3b975e7b822c3356ea7df153ca0a299392d3a650551cec0cb5395807bb"} Nov 03 00:16:22.164203 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.164197 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz" event={"ID":"84a4fdce-06fa-465d-95a3-092ba679cdea","Type":"ContainerDied","Data":"be87b6bc92642dd2f35395a40aeb81770a974700d8fc6b073389d7d145bae642"} Nov 03 00:16:22.164349 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.164236 6400 scope.go:122] "RemoveContainer" containerID="24973a3b975e7b822c3356ea7df153ca0a299392d3a650551cec0cb5395807bb" Nov 03 00:16:22.192662 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.192642 6400 scope.go:122] "RemoveContainer" containerID="24973a3b975e7b822c3356ea7df153ca0a299392d3a650551cec0cb5395807bb" Nov 03 00:16:22.193170 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:16:22.193152 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24973a3b975e7b822c3356ea7df153ca0a299392d3a650551cec0cb5395807bb\": container with ID starting with 24973a3b975e7b822c3356ea7df153ca0a299392d3a650551cec0cb5395807bb not found: ID does not exist" containerID="24973a3b975e7b822c3356ea7df153ca0a299392d3a650551cec0cb5395807bb" Nov 03 00:16:22.193255 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.193184 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24973a3b975e7b822c3356ea7df153ca0a299392d3a650551cec0cb5395807bb"} err="failed to get container status \"24973a3b975e7b822c3356ea7df153ca0a299392d3a650551cec0cb5395807bb\": rpc error: code = NotFound desc = could not find container \"24973a3b975e7b822c3356ea7df153ca0a299392d3a650551cec0cb5395807bb\": container with ID starting with 24973a3b975e7b822c3356ea7df153ca0a299392d3a650551cec0cb5395807bb not found: ID does not exist" Nov 03 00:16:22.215744 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.215706 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:22.325682 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.325647 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz"] Nov 03 00:16:22.329560 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.329535 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c6f27688-fdfd4b79b-dsbkz"] Nov 03 00:16:22.391815 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.391732 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:22.534692 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.534669 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf"] Nov 03 00:16:22.537855 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:16:22.537799 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0806992d_fe7f_49d4_80d8_78ceb1bccf72.slice/crio-4478d8e040d17531a11136f13855be9305b9c5fad42725848268d8b9ed73ad5f WatchSource:0}: Error finding container 4478d8e040d17531a11136f13855be9305b9c5fad42725848268d8b9ed73ad5f: Status 404 returned error can't find the container with id 4478d8e040d17531a11136f13855be9305b9c5fad42725848268d8b9ed73ad5f Nov 03 00:16:22.555239 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.555207 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:22.718549 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:22.718454 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:23.039960 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:23.039919 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:23.171082 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:23.171046 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" event={"ID":"0806992d-fe7f-49d4-80d8-78ceb1bccf72","Type":"ContainerStarted","Data":"4478d8e040d17531a11136f13855be9305b9c5fad42725848268d8b9ed73ad5f"} Nov 03 00:16:23.177677 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:23.177652 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a4fdce-06fa-465d-95a3-092ba679cdea" path="/var/lib/kubelet/pods/84a4fdce-06fa-465d-95a3-092ba679cdea/volumes" Nov 03 00:16:23.331977 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:23.331894 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:23.726142 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:23.726033 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:24.178346 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:24.178306 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" event={"ID":"0806992d-fe7f-49d4-80d8-78ceb1bccf72","Type":"ContainerStarted","Data":"bbe3b8f9e6ee177aa839f2a9e62aad0fef6a87406744e043bbedee2ea6601933"} Nov 03 00:16:24.178567 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:24.178413 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:24.293422 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:24.293368 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" podStartSLOduration=2.462566768 podStartE2EDuration="3.293352976s" totalImagesPullingTime="830.786208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:24.292641408 +0000 UTC m=+631.243581571" watchObservedRunningTime="2026-11-03 00:16:24.293352976 +0000 UTC m=+631.244293128" Nov 03 00:16:24.485184 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:24.485092 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:25.114138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:25.114097 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:25.417472 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:25.417379 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:27.151794 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:16:27.151757 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:16:27.449898 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:27.449795 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:29.025579 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:29.025544 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:29.266482 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:29.266447 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:29.446296 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:29.446260 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:30.187424 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:30.187398 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 00:16:32.237090 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:32.237006 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:16:32.484487 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:32.484451 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:32.640176 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:32.640136 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:34.013915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:34.013880 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:36.945764 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:36.945707 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:37.128151 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:37.128111 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:37.556038 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:37.556005 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:42.306530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:42.306491 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:44.599605 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:44.599567 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:44.776201 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:44.776157 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:47.094067 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:47.094033 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:48.058101 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:48.058069 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:16:49.420639 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.420602 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8"] Nov 03 00:16:49.421090 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.421031 6400 cpu_manager.go:414] "containerMap: removing container" podUID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" containerName="cleanup" Nov 03 00:16:49.421090 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.421044 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6d91d7-0b24-4b9e-954a-fe3b5e264b4e" containerName="cleanup" Nov 03 00:16:49.424310 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.424290 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.526959 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.526920 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-048135\"/\"kube-root-ca.crt\"" Nov 03 00:16:49.526959 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.526936 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-048135\"/\"openshift-service-ca.crt\"" Nov 03 00:16:49.526959 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.526954 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-048135\"/\"simulator-w4-048135-kserve-self-signed-certs\"" Nov 03 00:16:49.527205 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.526971 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-048135\"/\"default-dockercfg-xsf2g\"" Nov 03 00:16:49.547951 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.547923 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-dshm\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.548148 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.547959 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-home\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.548148 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.548042 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m24ql\" (UniqueName: \"kubernetes.io/projected/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-kube-api-access-m24ql\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.548148 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.548096 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-model-cache\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.548331 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.548172 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tls-certs\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.548331 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.548185 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:49.548331 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.548226 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tmp-dir\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.579244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.579155 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8"] Nov 03 00:16:49.649439 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.649384 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-dshm\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.649439 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.649439 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-home\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.649699 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.649462 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m24ql\" (UniqueName: \"kubernetes.io/projected/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-kube-api-access-m24ql\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.649699 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.649488 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-model-cache\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.649699 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.649507 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tls-certs\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.649699 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.649537 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tmp-dir\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.649951 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.649921 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-model-cache\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.650008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.649958 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-home\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.650008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.649976 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tmp-dir\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.651984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.651958 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-dshm\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.652130 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.652111 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tls-certs\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.706329 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.706256 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m24ql\" (UniqueName: \"kubernetes.io/projected/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-kube-api-access-m24ql\") pod \"simulator-w4-048135-kserve-54d59df6c8-6ntz8\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:49.736183 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:49.736137 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:50.110115 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:50.110082 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8"] Nov 03 00:16:50.112036 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:50.112005 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:50.112879 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:16:50.112816 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca16bbd6_e0a8_4d24_ba33_a749b8659aee.slice/crio-d253d311204662bb0309e52c00b88d9cdb6da4e928d4da1dce25f69bee9334fd WatchSource:0}: Error finding container d253d311204662bb0309e52c00b88d9cdb6da4e928d4da1dce25f69bee9334fd: Status 404 returned error can't find the container with id d253d311204662bb0309e52c00b88d9cdb6da4e928d4da1dce25f69bee9334fd Nov 03 00:16:50.114792 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:50.114776 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:16:50.299912 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:50.299870 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" event={"ID":"ca16bbd6-e0a8-4d24-ba33-a749b8659aee","Type":"ContainerStarted","Data":"d253d311204662bb0309e52c00b88d9cdb6da4e928d4da1dce25f69bee9334fd"} Nov 03 00:16:51.836261 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.836221 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr"] Nov 03 00:16:51.840467 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.840443 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:51.868962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.868930 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p4fg\" (UniqueName: \"kubernetes.io/projected/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-kube-api-access-5p4fg\") pod \"maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:51.869108 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.868989 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-metrics-certs\") pod \"maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:51.869180 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.869153 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:51.869222 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.869204 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:51.900060 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.900028 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-d7bfc2-serving-cert\"" Nov 03 00:16:51.947874 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.947816 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr"] Nov 03 00:16:51.972641 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.970654 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:51.972641 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.970715 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:51.972641 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.970762 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4fg\" (UniqueName: \"kubernetes.io/projected/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-kube-api-access-5p4fg\") pod \"maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:51.972641 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.970807 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-metrics-certs\") pod \"maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:51.972641 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:16:51.970865 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-serving-cert: secret "maas-api-e2e-worker-w3-d7bfc2-serving-cert" not found Nov 03 00:16:51.972641 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:16:51.970943 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-maas-api-tls podName:bc8bfcd4-3359-4a5f-aeef-50516cf73fa3 nodeName:}" failed. No retries permitted until 2026-11-03 00:16:52.470919274 +0000 UTC m=+659.421859415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-maas-api-tls") pod "maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" (UID: "bc8bfcd4-3359-4a5f-aeef-50516cf73fa3") : secret "maas-api-e2e-worker-w3-d7bfc2-serving-cert" not found Nov 03 00:16:51.972641 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.971489 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:51.977387 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:51.977358 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-metrics-certs\") pod \"maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:52.028762 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:52.028728 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p4fg\" (UniqueName: \"kubernetes.io/projected/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-kube-api-access-5p4fg\") pod \"maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:52.058720 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:52.058684 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:52.222300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:52.222219 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:52.471461 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:52.471430 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:52.476590 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:52.476501 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:52.485596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:52.485565 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:52.756468 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:52.756363 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:53.130561 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:53.130526 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr"] Nov 03 00:16:53.134162 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:16:53.134133 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8bfcd4_3359_4a5f_aeef_50516cf73fa3.slice/crio-00953047bb1ce9600072303b4844acc62ff5fabfdf2ff2858505309a2274c938 WatchSource:0}: Error finding container 00953047bb1ce9600072303b4844acc62ff5fabfdf2ff2858505309a2274c938: Status 404 returned error can't find the container with id 00953047bb1ce9600072303b4844acc62ff5fabfdf2ff2858505309a2274c938 Nov 03 00:16:53.318927 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:53.318881 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" event={"ID":"ca16bbd6-e0a8-4d24-ba33-a749b8659aee","Type":"ContainerStarted","Data":"cb88806e94d9d9aecc98cdbea30808ef778ff428c8f89f0e22282cbafe66c4ad"} Nov 03 00:16:53.320025 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:53.319998 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" event={"ID":"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3","Type":"ContainerStarted","Data":"00953047bb1ce9600072303b4844acc62ff5fabfdf2ff2858505309a2274c938"} Nov 03 00:16:53.434288 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:53.434178 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" podStartSLOduration=2.204701106 podStartE2EDuration="4.434161248s" totalImagesPullingTime="2.229460142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:53.432344877 +0000 UTC m=+660.383285049" watchObservedRunningTime="2026-11-03 00:16:53.434161248 +0000 UTC m=+660.385101405" Nov 03 00:16:53.736631 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:53.736545 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:53.825885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:53.825845 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:53.996992 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:53.996951 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:54.326302 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:54.326259 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" event={"ID":"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3","Type":"ContainerStarted","Data":"f507790e8e9a2c933e001e261c45ec5ba7c707981cc3d478b81d1f71fa7587b1"} Nov 03 00:16:54.326805 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:54.326347 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:16:54.440160 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:54.440110 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" podStartSLOduration=2.5987637120000002 podStartE2EDuration="3.440094541s" totalImagesPullingTime="841.330829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:54.438781178 +0000 UTC m=+661.389721336" watchObservedRunningTime="2026-11-03 00:16:54.440094541 +0000 UTC m=+661.391034698" Nov 03 00:16:56.754197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:56.753744 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:57.152544 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:16:57.152505 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:16:57.332510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:57.332467 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:57.611935 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:57.611901 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:59.737096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:59.737063 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:16:59.749861 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:16:59.749815 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:17:00.337507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:00.337477 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 00:17:00.367556 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:00.367526 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 00:17:01.810159 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:01.810124 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:02.116346 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:02.116259 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:02.116772 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:02.116753 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:17:06.801198 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:06.801167 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:11.914847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:11.914784 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:12.382272 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:12.382244 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:12.964043 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:12.964005 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:13.141119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:13.141077 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:14.446892 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:14.446848 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:16.702680 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:16.702645 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:19.490473 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.490438 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf"] Nov 03 00:17:19.494156 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.494139 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.549751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.549689 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e28e1a\"/\"openshift-service-ca.crt\"" Nov 03 00:17:19.549964 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.549795 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e28e1a\"/\"default-dockercfg-7v5bh\"" Nov 03 00:17:19.567390 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.567355 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e28e1a\"/\"kube-root-ca.crt\"" Nov 03 00:17:19.597158 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.597118 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e28e1a\"/\"simulator-w1-e28e1a-kserve-self-signed-certs\"" Nov 03 00:17:19.599159 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.599130 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf"] Nov 03 00:17:19.619903 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.619865 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:17:19.620751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.620722 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:19.650839 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.650791 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-home\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.650977 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.650876 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tls-certs\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.650977 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.650940 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9vnp\" (UniqueName: \"kubernetes.io/projected/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-kube-api-access-g9vnp\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.650977 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.650967 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tmp-dir\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.651080 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.650994 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-model-cache\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.651115 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.651082 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-dshm\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.752473 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.752379 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tmp-dir\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.752473 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.752434 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-model-cache\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.752698 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.752491 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-dshm\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.752698 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.752572 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-home\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.752698 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.752655 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tls-certs\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.752698 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.752694 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9vnp\" (UniqueName: \"kubernetes.io/projected/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-kube-api-access-g9vnp\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.752952 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.752815 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tmp-dir\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.753035 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.752993 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-model-cache\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.753234 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.753100 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-home\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.755514 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.755492 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-dshm\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.755989 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.755965 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tls-certs\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:19.814638 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:19.814606 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9vnp\" (UniqueName: \"kubernetes.io/projected/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-kube-api-access-g9vnp\") pod \"simulator-w1-e28e1a-kserve-6674b5c878-q6tpf\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:20.106218 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:20.106183 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:20.478541 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:20.478487 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf"] Nov 03 00:17:20.481324 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:17:20.481291 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f7d333a_0851_4b86_a3ab_29b6ea453fa7.slice/crio-82262f72200caa1d799bd73f01ec473bbd8c606e0bed5a8470e64fe774520548 WatchSource:0}: Error finding container 82262f72200caa1d799bd73f01ec473bbd8c606e0bed5a8470e64fe774520548: Status 404 returned error can't find the container with id 82262f72200caa1d799bd73f01ec473bbd8c606e0bed5a8470e64fe774520548 Nov 03 00:17:21.419283 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:21.419245 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:21.456713 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:21.456674 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" event={"ID":"1f7d333a-0851-4b86-a3ab-29b6ea453fa7","Type":"ContainerStarted","Data":"d8ecc944a8982053963b7e548c16d9afd082a0e15a9dd7ee5f9d7ceeb845358a"} Nov 03 00:17:21.456934 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:21.456720 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" event={"ID":"1f7d333a-0851-4b86-a3ab-29b6ea453fa7","Type":"ContainerStarted","Data":"82262f72200caa1d799bd73f01ec473bbd8c606e0bed5a8470e64fe774520548"} Nov 03 00:17:21.570671 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:21.570590 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" podStartSLOduration=2.570571803 podStartE2EDuration="2.570571803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:21.569205732 +0000 UTC m=+688.520145883" watchObservedRunningTime="2026-11-03 00:17:21.570571803 +0000 UTC m=+688.521511961" Nov 03 00:17:22.106473 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:22.106440 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:26.119377 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:26.119332 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:27.152392 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:17:27.152352 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:17:29.865004 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:29.864965 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:30.107324 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:30.107291 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:30.125667 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:30.121269 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:30.479024 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:30.478929 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:30.509295 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:30.509264 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 00:17:30.659789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:30.659753 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:31.031105 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:31.031075 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:31.872832 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:31.872787 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:33.228275 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:33.228239 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:17:35.896636 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:35.896599 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:40.650688 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:40.650650 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:44.887523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:44.887467 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:45.311997 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:45.311959 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:48.947124 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:48.947085 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:17:50.014508 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:50.014455 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:54.707966 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:54.707916 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:57.152395 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:17:57.152357 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:17:59.248474 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:59.248433 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:59.455104 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:17:59.455062 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:02.154571 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:02.154481 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:04.108347 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:04.108311 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:06.250002 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:06.249962 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:18:07.856262 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:07.856227 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g"] Nov 03 00:18:07.860054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:07.860033 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:07.964021 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:07.963952 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:07.964255 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:07.964143 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ebeeb9\"/\"openshift-service-ca.crt\"" Nov 03 00:18:07.964378 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:07.964266 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ebeeb9\"/\"premium-w0-ebeeb9-kserve-self-signed-certs\"" Nov 03 00:18:07.964378 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:07.964293 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ebeeb9\"/\"kube-root-ca.crt\"" Nov 03 00:18:07.964378 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:07.964304 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ebeeb9\"/\"default-dockercfg-64htb\"" Nov 03 00:18:07.965648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:07.965628 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g"] Nov 03 00:18:08.026713 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.026677 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jgkc\" (UniqueName: \"kubernetes.io/projected/89d21cbe-9619-4d67-bf52-105653fa5e73-kube-api-access-7jgkc\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.026713 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.026715 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-tmp-dir\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.026996 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.026744 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-dshm\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.026996 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.026760 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-home\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.026996 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.026897 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d21cbe-9619-4d67-bf52-105653fa5e73-tls-certs\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.026996 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.026996 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-model-cache\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.128488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.128388 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d21cbe-9619-4d67-bf52-105653fa5e73-tls-certs\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.128488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.128449 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-model-cache\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.128742 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.128580 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jgkc\" (UniqueName: \"kubernetes.io/projected/89d21cbe-9619-4d67-bf52-105653fa5e73-kube-api-access-7jgkc\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.128742 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.128611 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-tmp-dir\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.128742 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.128653 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-dshm\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.128742 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.128674 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-home\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.128978 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.128940 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-model-cache\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.129198 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.129177 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-home\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.129393 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.129281 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-tmp-dir\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.131072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.131048 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-dshm\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.131465 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.131445 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d21cbe-9619-4d67-bf52-105653fa5e73-tls-certs\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.185985 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.185941 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jgkc\" (UniqueName: \"kubernetes.io/projected/89d21cbe-9619-4d67-bf52-105653fa5e73-kube-api-access-7jgkc\") pod \"premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.471525 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.471426 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:08.847803 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.847604 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g"] Nov 03 00:18:08.850702 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:18:08.850677 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89d21cbe_9619_4d67_bf52_105653fa5e73.slice/crio-d6256a4085b50b54ebeb1bafb1b6377c3d6625303a1a2d45c069334b603ea7b0 WatchSource:0}: Error finding container d6256a4085b50b54ebeb1bafb1b6377c3d6625303a1a2d45c069334b603ea7b0: Status 404 returned error can't find the container with id d6256a4085b50b54ebeb1bafb1b6377c3d6625303a1a2d45c069334b603ea7b0 Nov 03 00:18:08.895711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:08.895680 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:09.695215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:09.695177 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" event={"ID":"89d21cbe-9619-4d67-bf52-105653fa5e73","Type":"ContainerStarted","Data":"f57be52941981ae797a6406a134245e78fb20e061eb0609ce3d4505d030263eb"} Nov 03 00:18:09.695215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:09.695223 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" event={"ID":"89d21cbe-9619-4d67-bf52-105653fa5e73","Type":"ContainerStarted","Data":"d6256a4085b50b54ebeb1bafb1b6377c3d6625303a1a2d45c069334b603ea7b0"} Nov 03 00:18:09.810847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:09.810775 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" podStartSLOduration=2.810761302 podStartE2EDuration="2.810761302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:09.808628257 +0000 UTC m=+736.759568414" watchObservedRunningTime="2026-11-03 00:18:09.810761302 +0000 UTC m=+736.761701460" Nov 03 00:18:10.471778 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:10.471744 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:13.596721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:13.596683 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:18.159861 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.159815 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs"] Nov 03 00:18:18.163470 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.163454 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.225219 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.225179 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxp52\" (UniqueName: \"kubernetes.io/projected/2fd1b7e8-1357-47d7-9ba6-236e95443aae-kube-api-access-cxp52\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.225386 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.225318 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-model-cache\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.225467 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.225444 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tls-certs\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.225567 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.225550 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tmp-dir\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.225614 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.225601 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-home\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.225649 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.225622 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-dshm\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.267216 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.267183 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e28e1a\"/\"premium-w1-e28e1a-kserve-self-signed-certs\"" Nov 03 00:18:18.268565 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.268545 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs"] Nov 03 00:18:18.282318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.282284 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:18.327011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.326980 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxp52\" (UniqueName: \"kubernetes.io/projected/2fd1b7e8-1357-47d7-9ba6-236e95443aae-kube-api-access-cxp52\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.327187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.327030 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-model-cache\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.327187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.327066 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tls-certs\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.327187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.327111 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tmp-dir\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.327187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.327149 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-home\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.327187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.327172 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-dshm\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.327565 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.327541 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-model-cache\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.327684 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.327547 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tmp-dir\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.327739 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.327681 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-home\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.329635 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.329614 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-dshm\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.329912 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.329895 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tls-certs\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.383866 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.383832 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxp52\" (UniqueName: \"kubernetes.io/projected/2fd1b7e8-1357-47d7-9ba6-236e95443aae-kube-api-access-cxp52\") pod \"premium-w1-e28e1a-kserve-759df99987-279zs\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.472412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.472330 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:18.476206 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.476183 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:18.491363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.491330 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:18.502480 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.502455 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:18.759014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.758938 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:18:18.859752 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:18.859634 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs"] Nov 03 00:18:18.862162 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:18:18.862127 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fd1b7e8_1357_47d7_9ba6_236e95443aae.slice/crio-00dfdc856a261d0f9bc44a775805bd0cc9f6b9da70bef02f4ecbcfc7c5b160f3 WatchSource:0}: Error finding container 00dfdc856a261d0f9bc44a775805bd0cc9f6b9da70bef02f4ecbcfc7c5b160f3: Status 404 returned error can't find the container with id 00dfdc856a261d0f9bc44a775805bd0cc9f6b9da70bef02f4ecbcfc7c5b160f3 Nov 03 00:18:19.752988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:19.752949 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" event={"ID":"2fd1b7e8-1357-47d7-9ba6-236e95443aae","Type":"ContainerStarted","Data":"cc44c5285df5662c6df7ba541c4d2b184c91a4c7e76fb1452fa79f7522e2d00a"} Nov 03 00:18:19.752988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:19.752995 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" event={"ID":"2fd1b7e8-1357-47d7-9ba6-236e95443aae","Type":"ContainerStarted","Data":"00dfdc856a261d0f9bc44a775805bd0cc9f6b9da70bef02f4ecbcfc7c5b160f3"} Nov 03 00:18:19.866255 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:19.866059 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" podStartSLOduration=1.866044907 podStartE2EDuration="1.866044907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:19.865521972 +0000 UTC m=+746.816462134" watchObservedRunningTime="2026-11-03 00:18:19.866044907 +0000 UTC m=+746.816985064" Nov 03 00:18:20.476438 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:20.476394 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:23.107960 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:23.107924 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:24.246887 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:24.246856 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:18:27.153152 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:18:27.153115 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:18:27.929886 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:27.929850 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:28.477374 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:28.477342 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:28.490488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:28.490456 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:28.809095 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:28.809064 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 00:18:32.603523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:32.603481 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:35.951191 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:35.951157 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:36.113937 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:36.113896 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:37.315452 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:37.315414 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:41.119079 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:41.119023 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:42.105948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:42.105908 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:46.328985 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:46.328947 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:46.814730 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:46.814701 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:47.941695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:47.941666 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:18:51.589214 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:51.589177 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:56.280759 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:18:56.280720 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:57.152792 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:18:57.152755 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:19:01.142055 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:01.142003 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:03.111969 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:03.111933 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:05.915007 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:05.914965 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:06.295112 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:06.295083 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:10.611506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:10.611473 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:12.941254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:12.941219 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:19:15.349751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:15.349714 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:20.101147 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:20.101111 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:24.803242 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:24.803198 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:27.154566 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:19:27.154528 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:19:29.501146 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:29.501117 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:34.253031 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:34.252953 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:37.209067 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:37.209029 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:38.928471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:38.928436 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:43.623117 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:43.623081 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:48.314503 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:48.314464 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:52.336093 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:52.336054 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:19:53.025551 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:53.025519 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:57.153664 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:19:57.153623 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:19:57.677592 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:19:57.677558 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:02.388154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:02.388117 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:02.837514 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:02.837479 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:05.786629 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:05.786595 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:06.022855 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:06.022808 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:07.097288 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:07.097240 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:11.840084 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:11.840044 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:12.493003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:12.492972 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:16.718142 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:16.718109 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:17.865267 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:17.865231 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:20:21.421242 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:21.421196 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:22.450036 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.450001 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n"] Nov 03 00:20:22.452606 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.452586 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.562969 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.562928 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.563140 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.562984 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.563140 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.563073 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8k8\" (UniqueName: \"kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-kube-api-access-zc8k8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.563140 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.563118 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.563260 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.563157 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.563260 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.563248 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.563334 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.563288 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.563334 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.563312 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.563407 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.563366 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd0c634-3b18-4b15-8438-6cf01db8b352-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.660565 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.660519 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n"] Nov 03 00:20:22.664507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.664479 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.664688 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.664540 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.664688 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.664604 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd0c634-3b18-4b15-8438-6cf01db8b352-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.664688 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.664657 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.664881 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.664701 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.664881 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.664724 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8k8\" (UniqueName: \"kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-kube-api-access-zc8k8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.664881 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.664761 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.664881 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.664790 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.665078 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.664900 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.665339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.665312 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.666065 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.665365 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.666065 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.665726 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.666256 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.666094 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd0c634-3b18-4b15-8438-6cf01db8b352-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.666315 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.666252 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.668106 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.668079 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.668340 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.668322 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.721734 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.721659 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.721893 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.721778 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8k8\" (UniqueName: \"kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-kube-api-access-zc8k8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rm28n\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:22.764037 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:22.764008 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:23.198904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.194746 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n"] Nov 03 00:20:23.201597 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:20:23.201564 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd0c634_3b18_4b15_8438_6cf01db8b352.slice/crio-024dde751a7e22ba9c0a481581bd32ca5f494b09327e1b882a4dbbdd5623ab8d WatchSource:0}: Error finding container 024dde751a7e22ba9c0a481581bd32ca5f494b09327e1b882a4dbbdd5623ab8d: Status 404 returned error can't find the container with id 024dde751a7e22ba9c0a481581bd32ca5f494b09327e1b882a4dbbdd5623ab8d Nov 03 00:20:23.203598 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.203562 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:20:23.203695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.203628 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:20:23.203695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.203657 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:20:23.351950 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.351908 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" event={"ID":"6dd0c634-3b18-4b15-8438-6cf01db8b352","Type":"ContainerStarted","Data":"f019f7cc6e5881b12d8a6d94f1429b84a237c2cfe9fd4a29e30c5ba0c55977f9"} Nov 03 00:20:23.352137 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.351984 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" event={"ID":"6dd0c634-3b18-4b15-8438-6cf01db8b352","Type":"ContainerStarted","Data":"024dde751a7e22ba9c0a481581bd32ca5f494b09327e1b882a4dbbdd5623ab8d"} Nov 03 00:20:23.432423 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.432391 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq"] Nov 03 00:20:23.434894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.434877 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.472643 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.472538 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" podStartSLOduration=1.472523787 podStartE2EDuration="1.472523787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:23.470951118 +0000 UTC m=+870.421891277" watchObservedRunningTime="2026-11-03 00:20:23.472523787 +0000 UTC m=+870.423463956" Nov 03 00:20:23.474644 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.474616 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-model-cache\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.474768 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.474661 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-tmp-dir\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.474831 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.474791 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-home\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.474905 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.474887 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/965b9206-0420-4092-a275-287beb605ffb-tls-certs\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.475015 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.474997 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf8dt\" (UniqueName: \"kubernetes.io/projected/965b9206-0420-4092-a275-287beb605ffb-kube-api-access-pf8dt\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.475063 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.475051 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-dshm\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.487908 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.487883 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"kube-root-ca.crt\"" Nov 03 00:20:23.488097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.488075 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"e2e-conflict-a-67ffbfa6-kserve-self-signed-certs\"" Nov 03 00:20:23.494345 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.494319 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq"] Nov 03 00:20:23.547696 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.547664 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:23.577842 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.577787 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf8dt\" (UniqueName: \"kubernetes.io/projected/965b9206-0420-4092-a275-287beb605ffb-kube-api-access-pf8dt\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.578144 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.578104 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-dshm\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.578274 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.578209 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-model-cache\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.578274 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.578259 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-tmp-dir\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.578388 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.578306 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-home\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.578388 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.578363 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/965b9206-0420-4092-a275-287beb605ffb-tls-certs\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.579179 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.579157 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-model-cache\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.579439 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.579377 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-tmp-dir\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.584654 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.579749 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-home\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.584807 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.581709 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/965b9206-0420-4092-a275-287beb605ffb-tls-certs\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.584996 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.582673 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-dshm\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.588137 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.588112 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"openshift-service-ca.crt\"" Nov 03 00:20:23.588224 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.588159 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"default-dockercfg-mpmcw\"" Nov 03 00:20:23.635680 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.635646 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf8dt\" (UniqueName: \"kubernetes.io/projected/965b9206-0420-4092-a275-287beb605ffb-kube-api-access-pf8dt\") pod \"e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.750560 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.750459 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:23.764583 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:23.764552 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:24.206380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:24.206314 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq"] Nov 03 00:20:24.208897 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:20:24.208870 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod965b9206_0420_4092_a275_287beb605ffb.slice/crio-f1594b04e0b90ab1cc2dffb7973550c6650f9e51291b56e98328f7ceab5f2d8c WatchSource:0}: Error finding container f1594b04e0b90ab1cc2dffb7973550c6650f9e51291b56e98328f7ceab5f2d8c: Status 404 returned error can't find the container with id f1594b04e0b90ab1cc2dffb7973550c6650f9e51291b56e98328f7ceab5f2d8c Nov 03 00:20:24.358801 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:24.358752 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" event={"ID":"965b9206-0420-4092-a275-287beb605ffb","Type":"ContainerStarted","Data":"e61021f0dbdacf99d901d7fc2a6e1a39beeab98d99bbc1db22ad04f01ae84136"} Nov 03 00:20:24.358801 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:24.358806 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" event={"ID":"965b9206-0420-4092-a275-287beb605ffb","Type":"ContainerStarted","Data":"f1594b04e0b90ab1cc2dffb7973550c6650f9e51291b56e98328f7ceab5f2d8c"} Nov 03 00:20:24.492932 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:24.491796 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" podStartSLOduration=1.491774206 podStartE2EDuration="1.491774206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:24.489269427 +0000 UTC m=+871.440209601" watchObservedRunningTime="2026-11-03 00:20:24.491774206 +0000 UTC m=+871.442714365" Nov 03 00:20:24.750750 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:24.750662 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:24.765844 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:24.765795 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 00:20:24.765962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:24.765878 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" podUID="6dd0c634-3b18-4b15-8438-6cf01db8b352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 00:20:25.764516 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:25.764472 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 00:20:25.765042 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:25.764543 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" podUID="6dd0c634-3b18-4b15-8438-6cf01db8b352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 00:20:26.214917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:26.214879 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:26.764922 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:26.764884 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 00:20:26.765324 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:26.764938 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" podUID="6dd0c634-3b18-4b15-8438-6cf01db8b352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 00:20:27.152949 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:20:27.152915 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:20:27.764694 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:27.764658 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 00:20:27.764896 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:27.764716 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" podUID="6dd0c634-3b18-4b15-8438-6cf01db8b352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 00:20:28.454549 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:28.454515 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:28.764952 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:28.764869 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 00:20:28.764952 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:28.764926 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" podUID="6dd0c634-3b18-4b15-8438-6cf01db8b352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 00:20:29.764981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:29.764943 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 00:20:29.765413 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:29.765003 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" podUID="6dd0c634-3b18-4b15-8438-6cf01db8b352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 00:20:30.765126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:30.765093 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 00:20:30.765543 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:30.765163 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" podUID="6dd0c634-3b18-4b15-8438-6cf01db8b352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 00:20:31.027444 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:31.027353 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:31.769709 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:31.769675 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:31.770196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:31.770075 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:31.770670 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:31.770653 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:20:33.751226 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:33.751198 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:33.770499 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:33.770467 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:34.446709 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:34.446676 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:20:35.858320 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:35.858282 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:36.025756 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:36.025708 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:37.566135 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.566102 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw"] Nov 03 00:20:37.569794 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.569774 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.653605 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.653570 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.653769 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.653626 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.653769 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.653722 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.653876 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.653765 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.653876 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.653815 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj6fg\" (UniqueName: \"kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-kube-api-access-rj6fg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.653942 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.653890 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.653942 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.653915 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.654006 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.653991 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.654080 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.654067 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.755238 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.755205 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.755410 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.755276 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.755410 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.755339 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.755410 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.755364 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.755410 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.755400 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.755618 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.755433 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6fg\" (UniqueName: \"kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-kube-api-access-rj6fg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.755618 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.755460 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.755618 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.755493 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.755618 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.755533 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.755809 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.755751 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.755962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.755934 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.756194 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.756068 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.756194 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.756155 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.756445 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.756424 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.757834 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.757796 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.758555 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.758536 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.779099 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.779074 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw"] Nov 03 00:20:37.811584 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.811556 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.811774 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.811754 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj6fg\" (UniqueName: \"kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-kube-api-access-rj6fg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-svljw\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:37.883980 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:37.883889 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:38.312337 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:38.312313 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw"] Nov 03 00:20:38.315044 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:20:38.315008 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a52a9c6_c4a0_4fb1_a682_cbc3c279aedf.slice/crio-879367b800fbd573ee44565c07e7d925f8cd5c8f09368392f09fa6fd15d0add4 WatchSource:0}: Error finding container 879367b800fbd573ee44565c07e7d925f8cd5c8f09368392f09fa6fd15d0add4: Status 404 returned error can't find the container with id 879367b800fbd573ee44565c07e7d925f8cd5c8f09368392f09fa6fd15d0add4 Nov 03 00:20:38.317423 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:38.317386 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:20:38.317517 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:38.317473 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:20:38.317517 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:38.317505 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:20:38.462238 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:38.462198 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" event={"ID":"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf","Type":"ContainerStarted","Data":"6195135f9b7f24909ce27c8c217d0e950927c6d4aac3ddd2af33ae7358e47f7d"} Nov 03 00:20:38.462442 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:38.462268 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" event={"ID":"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf","Type":"ContainerStarted","Data":"879367b800fbd573ee44565c07e7d925f8cd5c8f09368392f09fa6fd15d0add4"} Nov 03 00:20:38.580475 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:38.580372 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" podStartSLOduration=1.580355529 podStartE2EDuration="1.580355529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:38.579705466 +0000 UTC m=+885.530645625" watchObservedRunningTime="2026-11-03 00:20:38.580355529 +0000 UTC m=+885.531295687" Nov 03 00:20:38.899954 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:38.891938 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:39.901039 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:39.900996 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-svljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 00:20:39.901467 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:39.901076 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 00:20:40.638426 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:40.638386 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:40.895910 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:40.893252 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-svljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 00:20:40.895910 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:40.893329 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 00:20:41.886848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:41.886242 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-svljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 00:20:41.886848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:41.886310 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 00:20:42.897301 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:42.897255 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-svljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 00:20:42.897978 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:42.897932 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 00:20:43.397848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:43.384813 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:20:43.892290 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:43.892250 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-svljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 00:20:43.892596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:43.892566 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 00:20:44.885318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:44.885276 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-svljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 00:20:44.885743 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:44.885351 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 00:20:45.436511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:45.436473 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:45.885702 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:45.885238 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-svljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 00:20:45.885702 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:45.885306 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 00:20:47.884689 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:47.884646 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-svljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:47.885227 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:47.884714 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:48.926038 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:48.925981 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-svljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:48.926589 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:48.926061 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:49.083944 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:49.083909 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:49.084744 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:49.084718 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:49.088334 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:49.088314 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:20:50.221917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:50.221878 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:51.951503 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:51.951471 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:52.120716 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:52.120681 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:54.907509 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:54.907466 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:57.151517 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:20:57.151478 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:20:57.154651 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.154625 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z"] Nov 03 00:20:57.159449 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.159430 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.211269 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.211222 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d7bfc2-openshift-default-dockercfg-g56sm\"" Nov 03 00:20:57.263434 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.263395 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z"] Nov 03 00:20:57.298272 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.298239 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42973673-96bf-4c81-9b1d-570456790ee3-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.298422 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.298275 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.298497 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.298475 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.298542 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.298527 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.298581 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.298568 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.298656 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.298643 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.298703 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.298671 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgt89\" (UniqueName: \"kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-kube-api-access-pgt89\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.298745 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.298724 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.298780 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.298754 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42973673-96bf-4c81-9b1d-570456790ee3-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.399356 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.399324 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42973673-96bf-4c81-9b1d-570456790ee3-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.399356 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.399361 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.399571 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.399477 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.399571 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.399516 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.399571 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.399544 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.399708 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.399591 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.399708 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.399617 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgt89\" (UniqueName: \"kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-kube-api-access-pgt89\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.399708 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.399661 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.399708 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.399704 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42973673-96bf-4c81-9b1d-570456790ee3-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.399931 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.399749 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.399931 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.399787 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.400033 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.399927 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.400033 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.399981 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.400667 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.400634 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42973673-96bf-4c81-9b1d-570456790ee3-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.402158 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.402105 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.402256 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.402224 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42973673-96bf-4c81-9b1d-570456790ee3-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.457070 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.457044 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.457256 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.457233 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgt89\" (UniqueName: \"kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-kube-api-access-pgt89\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.472478 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.472438 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:57.896422 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.896394 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z"] Nov 03 00:20:57.899032 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:20:57.899005 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42973673_96bf_4c81_9b1d_570456790ee3.slice/crio-ab6598fd6516f37afe1828c3e0757f5bffb046268947b7c889487b7cae470c0d WatchSource:0}: Error finding container ab6598fd6516f37afe1828c3e0757f5bffb046268947b7c889487b7cae470c0d: Status 404 returned error can't find the container with id ab6598fd6516f37afe1828c3e0757f5bffb046268947b7c889487b7cae470c0d Nov 03 00:20:57.901435 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.901403 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:20:57.901512 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.901473 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:20:57.901512 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:57.901502 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:20:58.613307 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:58.613264 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" event={"ID":"42973673-96bf-4c81-9b1d-570456790ee3","Type":"ContainerStarted","Data":"9aef367b02ce5292c2b51208d6b50210ad87edeb3147c0eedb2fecfd9ca0ee1f"} Nov 03 00:20:58.613757 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:58.613333 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" event={"ID":"42973673-96bf-4c81-9b1d-570456790ee3","Type":"ContainerStarted","Data":"ab6598fd6516f37afe1828c3e0757f5bffb046268947b7c889487b7cae470c0d"} Nov 03 00:20:58.779387 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:58.779312 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" podStartSLOduration=1.779293134 podStartE2EDuration="1.779293134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:58.77779775 +0000 UTC m=+905.728737908" watchObservedRunningTime="2026-11-03 00:20:58.779293134 +0000 UTC m=+905.730233290" Nov 03 00:20:59.451588 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:59.451550 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:59.473243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:59.473214 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:20:59.474615 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:59.474587 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 00:20:59.474776 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:59.474637 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" podUID="42973673-96bf-4c81-9b1d-570456790ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 00:20:59.662969 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:20:59.662915 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:00.473357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:00.473322 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 00:21:00.473568 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:00.473383 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" podUID="42973673-96bf-4c81-9b1d-570456790ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 00:21:01.473675 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:01.473577 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 00:21:01.474203 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:01.473678 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" podUID="42973673-96bf-4c81-9b1d-570456790ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 00:21:02.473741 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:02.473696 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 00:21:02.474420 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:02.474378 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" podUID="42973673-96bf-4c81-9b1d-570456790ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 00:21:03.332363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:03.332325 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:03.487851 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:03.484118 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 00:21:03.487851 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:03.484181 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" podUID="42973673-96bf-4c81-9b1d-570456790ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 00:21:03.885145 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:03.885083 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:21:03.885499 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:03.885440 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" podUID="956c60bc-c036-450b-85d9-ffd5f5a0971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:21:04.475391 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:04.475339 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 00:21:04.475722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:04.475671 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" podUID="42973673-96bf-4c81-9b1d-570456790ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 00:21:04.891047 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:04.891014 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:05.477617 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:05.477586 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:21:05.477884 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:05.477858 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:21:05.478565 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:05.478549 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:21:08.816146 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:08.816112 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:09.612384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:09.612350 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:14.404366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:14.404325 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:15.584207 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:15.584173 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:21:19.206515 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:19.206480 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:23.920488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:23.920433 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:27.141530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.141496 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp"] Nov 03 00:21:27.145527 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.145500 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.153254 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:21:27.153222 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:21:27.192758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.192723 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-dshm\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.193284 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.192785 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-home\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.193284 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.192922 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tmp-dir\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.193284 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.192972 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tls-certs\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.193284 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.192988 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr8cq\" (UniqueName: \"kubernetes.io/projected/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-kube-api-access-tr8cq\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.193284 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.193027 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-model-cache\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.250182 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.250150 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"e2e-conflict-b-67ffbfa6-kserve-self-signed-certs\"" Nov 03 00:21:27.252064 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.252043 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp"] Nov 03 00:21:27.258059 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.258028 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:27.326933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.326887 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tls-certs\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.327888 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.327813 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr8cq\" (UniqueName: \"kubernetes.io/projected/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-kube-api-access-tr8cq\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.328048 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.327938 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-model-cache\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.328167 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.328145 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-dshm\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.328239 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.328214 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-home\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.328293 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.328284 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tmp-dir\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.335840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.331584 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-home\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.335840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.331855 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tmp-dir\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.335840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.332003 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-model-cache\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.342005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.341971 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-dshm\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.342200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.342183 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tls-certs\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.387961 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.387932 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr8cq\" (UniqueName: \"kubernetes.io/projected/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-kube-api-access-tr8cq\") pod \"e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.459529 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.459442 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:27.896402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:27.896379 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp"] Nov 03 00:21:27.898882 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:21:27.898857 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod323c2ad9_ec85_440c_9f4a_fd12f3c9380d.slice/crio-ec2210b801a9838379288732ed00dea6ff8a3c3202f9c4d11560e48141028351 WatchSource:0}: Error finding container ec2210b801a9838379288732ed00dea6ff8a3c3202f9c4d11560e48141028351: Status 404 returned error can't find the container with id ec2210b801a9838379288732ed00dea6ff8a3c3202f9c4d11560e48141028351 Nov 03 00:21:28.712914 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:28.712870 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:28.777196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:28.777162 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" event={"ID":"323c2ad9-ec85-440c-9f4a-fd12f3c9380d","Type":"ContainerStarted","Data":"4d57f5fd39f32c9faee1741183db387f313843d2dd9edca5da257c2a580fa9ce"} Nov 03 00:21:28.777388 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:28.777207 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" event={"ID":"323c2ad9-ec85-440c-9f4a-fd12f3c9380d","Type":"ContainerStarted","Data":"ec2210b801a9838379288732ed00dea6ff8a3c3202f9c4d11560e48141028351"} Nov 03 00:21:28.891303 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:28.891249 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" podStartSLOduration=1.891235466 podStartE2EDuration="1.891235466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:28.890458413 +0000 UTC m=+935.841398584" watchObservedRunningTime="2026-11-03 00:21:28.891235466 +0000 UTC m=+935.842175648" Nov 03 00:21:29.459836 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:29.459789 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:33.520474 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:33.520436 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:37.179159 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:37.179132 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:21:37.459774 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:37.459685 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:37.471766 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:37.471733 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:37.833582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:37.833551 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:21:38.365904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:38.365869 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:41.530846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:41.530785 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:41.747723 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:41.747689 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:41.912318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:41.912289 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:43.035529 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:43.035495 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:47.818552 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:47.818519 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:52.827774 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:52.827737 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:57.152566 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:21:57.152530 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:21:57.699623 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:21:57.699587 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:00.065286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.065245 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h"] Nov 03 00:22:00.075406 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.075376 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.086102 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.086076 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h"] Nov 03 00:22:00.127686 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.127657 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-048135\"/\"distinct-2-w4-048135-kserve-self-signed-certs\"" Nov 03 00:22:00.138251 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.138209 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:00.142370 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.142349 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-tmp-dir\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.142437 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.142410 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-dshm\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.142537 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.142518 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b847w\" (UniqueName: \"kubernetes.io/projected/50ce1488-84fe-4013-970a-83af8bea40b1-kube-api-access-b847w\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.142580 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.142550 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-home\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.142642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.142624 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-model-cache\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.142696 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.142685 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ce1488-84fe-4013-970a-83af8bea40b1-tls-certs\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.244230 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.244197 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-dshm\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.244432 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.244358 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b847w\" (UniqueName: \"kubernetes.io/projected/50ce1488-84fe-4013-970a-83af8bea40b1-kube-api-access-b847w\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.244432 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.244415 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-home\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.244550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.244467 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-model-cache\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.244550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.244540 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ce1488-84fe-4013-970a-83af8bea40b1-tls-certs\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.244910 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.244879 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-tmp-dir\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.245028 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.244907 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-home\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.245028 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.244920 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-model-cache\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.245173 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.245152 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-tmp-dir\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.247275 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.247249 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ce1488-84fe-4013-970a-83af8bea40b1-tls-certs\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.247376 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.247306 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-dshm\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.301060 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.301022 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b847w\" (UniqueName: \"kubernetes.io/projected/50ce1488-84fe-4013-970a-83af8bea40b1-kube-api-access-b847w\") pod \"distinct-2-w4-048135-kserve-55948649d5-s954h\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.394898 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.391086 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:00.817580 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.817545 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h"] Nov 03 00:22:00.979400 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.979361 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" event={"ID":"50ce1488-84fe-4013-970a-83af8bea40b1","Type":"ContainerStarted","Data":"3c25df44b88b64473a1fa43ccd0cc3eb53beaa3b3b3ed178fad8a805d7082ae2"} Nov 03 00:22:00.979581 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:00.979411 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" event={"ID":"50ce1488-84fe-4013-970a-83af8bea40b1","Type":"ContainerStarted","Data":"d79c1ce23bfa69d5b1188aa6abf7d808a43c1a8baf9df2be3173691985c6f024"} Nov 03 00:22:01.093870 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:01.093750 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" podStartSLOduration=2.093735843 podStartE2EDuration="2.093735843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:01.092445883 +0000 UTC m=+968.043386040" watchObservedRunningTime="2026-11-03 00:22:01.093735843 +0000 UTC m=+968.044676001" Nov 03 00:22:01.392360 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:01.392277 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:02.443281 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:02.443248 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:03.632530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:03.632497 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:07.169679 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:07.169640 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:07.316789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:07.316743 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:22:10.397212 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:10.397177 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:10.462023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:10.461992 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:11.076286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:11.076256 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 00:22:12.012891 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:12.012857 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:16.724161 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:16.724125 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:18.217652 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:18.217616 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:21.443193 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:21.443156 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:26.225971 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:26.225914 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:27.153730 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:22:27.153684 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:22:29.428137 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:29.428103 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:31.034109 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:31.034076 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:35.712171 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:35.712133 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:36.988509 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:36.988477 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:22:40.511592 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:40.511562 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:45.116210 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:45.116176 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:49.828340 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:49.828300 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:53.024541 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:53.024495 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:54.608637 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:54.608601 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:55.135278 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:55.135246 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp"] Nov 03 00:22:55.172368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:55.172327 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:57.153980 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:22:57.153935 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:22:59.302174 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:22:59.302137 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:01.050834 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:01.050798 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq"] Nov 03 00:23:01.088161 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:01.088130 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:03.971471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:03.971435 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:04.137461 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:04.137416 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:07.732058 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:07.732024 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:23:08.906484 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:08.906446 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:10.157161 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.157092 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" podUID="323c2ad9-ec85-440c-9f4a-fd12f3c9380d" containerName="main" containerID="cri-o://4d57f5fd39f32c9faee1741183db387f313843d2dd9edca5da257c2a580fa9ce" gracePeriod=45 Nov 03 00:23:10.416613 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.416546 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:23:10.418451 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.418423 6400 generic.go:356] "Generic (PLEG): container finished" podID="323c2ad9-ec85-440c-9f4a-fd12f3c9380d" containerID="4d57f5fd39f32c9faee1741183db387f313843d2dd9edca5da257c2a580fa9ce" exitCode=0 Nov 03 00:23:10.418594 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.418466 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" Nov 03 00:23:10.418594 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.418516 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" event={"ID":"323c2ad9-ec85-440c-9f4a-fd12f3c9380d","Type":"ContainerDied","Data":"4d57f5fd39f32c9faee1741183db387f313843d2dd9edca5da257c2a580fa9ce"} Nov 03 00:23:10.418594 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.418573 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp" event={"ID":"323c2ad9-ec85-440c-9f4a-fd12f3c9380d","Type":"ContainerDied","Data":"ec2210b801a9838379288732ed00dea6ff8a3c3202f9c4d11560e48141028351"} Nov 03 00:23:10.418750 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.418596 6400 scope.go:122] "RemoveContainer" containerID="4d57f5fd39f32c9faee1741183db387f313843d2dd9edca5da257c2a580fa9ce" Nov 03 00:23:10.427555 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.427541 6400 scope.go:122] "RemoveContainer" containerID="4d57f5fd39f32c9faee1741183db387f313843d2dd9edca5da257c2a580fa9ce" Nov 03 00:23:10.427959 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:23:10.427934 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d57f5fd39f32c9faee1741183db387f313843d2dd9edca5da257c2a580fa9ce\": container with ID starting with 4d57f5fd39f32c9faee1741183db387f313843d2dd9edca5da257c2a580fa9ce not found: ID does not exist" containerID="4d57f5fd39f32c9faee1741183db387f313843d2dd9edca5da257c2a580fa9ce" Nov 03 00:23:10.428070 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.427968 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d57f5fd39f32c9faee1741183db387f313843d2dd9edca5da257c2a580fa9ce"} err="failed to get container status \"4d57f5fd39f32c9faee1741183db387f313843d2dd9edca5da257c2a580fa9ce\": rpc error: code = NotFound desc = could not find container \"4d57f5fd39f32c9faee1741183db387f313843d2dd9edca5da257c2a580fa9ce\": container with ID starting with 4d57f5fd39f32c9faee1741183db387f313843d2dd9edca5da257c2a580fa9ce not found: ID does not exist" Nov 03 00:23:10.562274 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.562241 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-model-cache\") pod \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " Nov 03 00:23:10.562459 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.562290 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-dshm\") pod \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " Nov 03 00:23:10.562459 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.562322 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-kube-api-access-tr8cq\" (UniqueName: \"kubernetes.io/projected/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-kube-api-access-tr8cq\") pod \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " Nov 03 00:23:10.562459 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.562402 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-home\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-home\") pod \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " Nov 03 00:23:10.562459 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.562434 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tls-certs\" (UniqueName: \"kubernetes.io/secret/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tls-certs\") pod \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " Nov 03 00:23:10.562654 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.562518 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tmp-dir\") pod \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\" (UID: \"323c2ad9-ec85-440c-9f4a-fd12f3c9380d\") " Nov 03 00:23:10.562654 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.562584 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-model-cache" pod "323c2ad9-ec85-440c-9f4a-fd12f3c9380d" (UID: "323c2ad9-ec85-440c-9f4a-fd12f3c9380d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:23:10.562871 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.562848 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-home" pod "323c2ad9-ec85-440c-9f4a-fd12f3c9380d" (UID: "323c2ad9-ec85-440c-9f4a-fd12f3c9380d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:23:10.563035 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.562887 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tmp-dir" pod "323c2ad9-ec85-440c-9f4a-fd12f3c9380d" (UID: "323c2ad9-ec85-440c-9f4a-fd12f3c9380d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:23:10.563336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.563121 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:10.563336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.563140 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:10.563336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.563154 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:10.565083 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.565061 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-dshm" pod "323c2ad9-ec85-440c-9f4a-fd12f3c9380d" (UID: "323c2ad9-ec85-440c-9f4a-fd12f3c9380d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:23:10.565322 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.565299 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tls-certs" pod "323c2ad9-ec85-440c-9f4a-fd12f3c9380d" (UID: "323c2ad9-ec85-440c-9f4a-fd12f3c9380d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:23:10.565381 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.565316 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-kube-api-access-tr8cq" pod "323c2ad9-ec85-440c-9f4a-fd12f3c9380d" (UID: "323c2ad9-ec85-440c-9f4a-fd12f3c9380d"). InnerVolumeSpecName "kube-api-access-tr8cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:23:10.664354 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.664324 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:10.664354 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.664352 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr8cq\" (UniqueName: \"kubernetes.io/projected/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-kube-api-access-tr8cq\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:10.664539 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.664366 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/323c2ad9-ec85-440c-9f4a-fd12f3c9380d-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:10.837091 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.837062 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp"] Nov 03 00:23:10.841570 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:10.841538 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-b-67ffbfa6-kserve-549c48cdb5-zhplp"] Nov 03 00:23:11.178760 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:11.178724 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323c2ad9-ec85-440c-9f4a-fd12f3c9380d" path="/var/lib/kubelet/pods/323c2ad9-ec85-440c-9f4a-fd12f3c9380d/volumes" Nov 03 00:23:13.609668 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:13.609636 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:16.072011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.071961 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" podUID="965b9206-0420-4092-a275-287beb605ffb" containerName="main" containerID="cri-o://e61021f0dbdacf99d901d7fc2a6e1a39beeab98d99bbc1db22ad04f01ae84136" gracePeriod=45 Nov 03 00:23:16.321290 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.321266 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:23:16.421082 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.420997 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-tmp-dir\") pod \"965b9206-0420-4092-a275-287beb605ffb\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " Nov 03 00:23:16.421082 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.421058 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965b9206-0420-4092-a275-287beb605ffb-kube-api-access-pf8dt\" (UniqueName: \"kubernetes.io/projected/965b9206-0420-4092-a275-287beb605ffb-kube-api-access-pf8dt\") pod \"965b9206-0420-4092-a275-287beb605ffb\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " Nov 03 00:23:16.421307 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.421132 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-home\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-home\") pod \"965b9206-0420-4092-a275-287beb605ffb\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " Nov 03 00:23:16.421307 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.421189 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-model-cache\") pod \"965b9206-0420-4092-a275-287beb605ffb\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " Nov 03 00:23:16.421307 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.421225 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-dshm\") pod \"965b9206-0420-4092-a275-287beb605ffb\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " Nov 03 00:23:16.421307 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.421239 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-tmp-dir" pod "965b9206-0420-4092-a275-287beb605ffb" (UID: "965b9206-0420-4092-a275-287beb605ffb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:23:16.421551 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.421310 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/965b9206-0420-4092-a275-287beb605ffb-tls-certs\" (UniqueName: \"kubernetes.io/secret/965b9206-0420-4092-a275-287beb605ffb-tls-certs\") pod \"965b9206-0420-4092-a275-287beb605ffb\" (UID: \"965b9206-0420-4092-a275-287beb605ffb\") " Nov 03 00:23:16.421551 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.421443 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-model-cache" pod "965b9206-0420-4092-a275-287beb605ffb" (UID: "965b9206-0420-4092-a275-287beb605ffb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:23:16.421551 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.421456 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-home" pod "965b9206-0420-4092-a275-287beb605ffb" (UID: "965b9206-0420-4092-a275-287beb605ffb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:23:16.421759 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.421742 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:16.421841 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.421765 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:16.421841 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.421782 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:16.423376 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.423357 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-dshm" pod "965b9206-0420-4092-a275-287beb605ffb" (UID: "965b9206-0420-4092-a275-287beb605ffb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:23:16.423435 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.423416 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965b9206-0420-4092-a275-287beb605ffb-kube-api-access-pf8dt" pod "965b9206-0420-4092-a275-287beb605ffb" (UID: "965b9206-0420-4092-a275-287beb605ffb"). InnerVolumeSpecName "kube-api-access-pf8dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:23:16.423488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.423472 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/965b9206-0420-4092-a275-287beb605ffb-tls-certs" pod "965b9206-0420-4092-a275-287beb605ffb" (UID: "965b9206-0420-4092-a275-287beb605ffb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:23:16.458862 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.458809 6400 generic.go:356] "Generic (PLEG): container finished" podID="965b9206-0420-4092-a275-287beb605ffb" containerID="e61021f0dbdacf99d901d7fc2a6e1a39beeab98d99bbc1db22ad04f01ae84136" exitCode=0 Nov 03 00:23:16.459003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.458881 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" Nov 03 00:23:16.459003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.458896 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" event={"ID":"965b9206-0420-4092-a275-287beb605ffb","Type":"ContainerDied","Data":"e61021f0dbdacf99d901d7fc2a6e1a39beeab98d99bbc1db22ad04f01ae84136"} Nov 03 00:23:16.459003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.458942 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq" event={"ID":"965b9206-0420-4092-a275-287beb605ffb","Type":"ContainerDied","Data":"f1594b04e0b90ab1cc2dffb7973550c6650f9e51291b56e98328f7ceab5f2d8c"} Nov 03 00:23:16.459003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.458960 6400 scope.go:122] "RemoveContainer" containerID="e61021f0dbdacf99d901d7fc2a6e1a39beeab98d99bbc1db22ad04f01ae84136" Nov 03 00:23:16.468843 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.468799 6400 scope.go:122] "RemoveContainer" containerID="e61021f0dbdacf99d901d7fc2a6e1a39beeab98d99bbc1db22ad04f01ae84136" Nov 03 00:23:16.469152 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:23:16.469135 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61021f0dbdacf99d901d7fc2a6e1a39beeab98d99bbc1db22ad04f01ae84136\": container with ID starting with e61021f0dbdacf99d901d7fc2a6e1a39beeab98d99bbc1db22ad04f01ae84136 not found: ID does not exist" containerID="e61021f0dbdacf99d901d7fc2a6e1a39beeab98d99bbc1db22ad04f01ae84136" Nov 03 00:23:16.469228 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.469162 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61021f0dbdacf99d901d7fc2a6e1a39beeab98d99bbc1db22ad04f01ae84136"} err="failed to get container status \"e61021f0dbdacf99d901d7fc2a6e1a39beeab98d99bbc1db22ad04f01ae84136\": rpc error: code = NotFound desc = could not find container \"e61021f0dbdacf99d901d7fc2a6e1a39beeab98d99bbc1db22ad04f01ae84136\": container with ID starting with e61021f0dbdacf99d901d7fc2a6e1a39beeab98d99bbc1db22ad04f01ae84136 not found: ID does not exist" Nov 03 00:23:16.523097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.523074 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf8dt\" (UniqueName: \"kubernetes.io/projected/965b9206-0420-4092-a275-287beb605ffb-kube-api-access-pf8dt\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:16.523097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.523097 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/965b9206-0420-4092-a275-287beb605ffb-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:16.523097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.523108 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/965b9206-0420-4092-a275-287beb605ffb-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:16.622016 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.621979 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq"] Nov 03 00:23:16.625476 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:16.625448 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/e2e-conflict-a-67ffbfa6-kserve-6c59455f6c-46qcq"] Nov 03 00:23:17.178685 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:17.178643 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="965b9206-0420-4092-a275-287beb605ffb" path="/var/lib/kubelet/pods/965b9206-0420-4092-a275-287beb605ffb/volumes" Nov 03 00:23:18.355932 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:18.355900 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:23.117220 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:23.117186 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:27.152206 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:23:27.152165 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:23:27.798336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:27.798298 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:28.294633 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.294597 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b"] Nov 03 00:23:28.295119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.295104 6400 cpu_manager.go:414] "containerMap: removing container" podUID="965b9206-0420-4092-a275-287beb605ffb" containerName="main" Nov 03 00:23:28.295163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.295122 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965b9206-0420-4092-a275-287beb605ffb" containerName="main" Nov 03 00:23:28.295163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.295133 6400 cpu_manager.go:414] "containerMap: removing container" podUID="323c2ad9-ec85-440c-9f4a-fd12f3c9380d" containerName="main" Nov 03 00:23:28.295163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.295139 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323c2ad9-ec85-440c-9f4a-fd12f3c9380d" containerName="main" Nov 03 00:23:28.295249 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.295201 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="965b9206-0420-4092-a275-287beb605ffb" containerName="main" Nov 03 00:23:28.295249 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.295209 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="323c2ad9-ec85-440c-9f4a-fd12f3c9380d" containerName="main" Nov 03 00:23:28.306578 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.306551 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.356582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.356541 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b"] Nov 03 00:23:28.455095 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.455055 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.455273 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.455109 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7mm\" (UniqueName: \"kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-kube-api-access-vb7mm\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.455273 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.455169 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.455273 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.455241 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.455419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.455277 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.455419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.455382 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.455491 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.455461 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.455555 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.455538 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c6e0c4-cc76-4dff-8073-8c399e461426-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.455599 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.455577 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.556642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.556548 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c6e0c4-cc76-4dff-8073-8c399e461426-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.556642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.556592 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.556969 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.556662 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.556969 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.556702 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7mm\" (UniqueName: \"kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-kube-api-access-vb7mm\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.556969 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.556783 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.556969 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.556883 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.556969 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.556913 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.557227 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.557075 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.557227 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.557151 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.557344 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.557324 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c6e0c4-cc76-4dff-8073-8c399e461426-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.557594 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.557577 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.557688 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.557667 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.557786 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.557768 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.557872 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.557804 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.559706 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.559684 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.559949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.559930 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.613454 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.613428 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.613694 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.613671 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7mm\" (UniqueName: \"kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-kube-api-access-vb7mm\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:28.622447 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:28.622420 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:29.116333 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:29.116268 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:23:29.117191 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:29.117130 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:23:29.117448 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:29.117424 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:23:29.118121 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:29.117197 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b"] Nov 03 00:23:29.528211 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:29.528122 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" event={"ID":"f9c6e0c4-cc76-4dff-8073-8c399e461426","Type":"ContainerStarted","Data":"7d0ee12727be3e88474b8566d217079ac7842502bf70d21e7163458232f9a9d7"} Nov 03 00:23:29.528211 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:29.528207 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" event={"ID":"f9c6e0c4-cc76-4dff-8073-8c399e461426","Type":"ContainerStarted","Data":"8ce289b8bc5aa625e39ecaf1865826957e6dc9bb1c45fc2c0189b4d9580a186f"} Nov 03 00:23:29.623405 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:29.623367 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:29.645368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:29.645298 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" podStartSLOduration=1.645278638 podStartE2EDuration="1.645278638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:29.644085744 +0000 UTC m=+1056.595025901" watchObservedRunningTime="2026-11-03 00:23:29.645278638 +0000 UTC m=+1056.596218801" Nov 03 00:23:30.624924 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:30.624885 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 00:23:30.625321 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:30.624951 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" podUID="f9c6e0c4-cc76-4dff-8073-8c399e461426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 00:23:31.623255 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:31.623221 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 00:23:31.623441 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:31.623284 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" podUID="f9c6e0c4-cc76-4dff-8073-8c399e461426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 00:23:32.548909 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:32.548871 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:32.630885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:32.625224 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 00:23:32.630885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:32.625294 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" podUID="f9c6e0c4-cc76-4dff-8073-8c399e461426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 00:23:33.623653 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:33.623623 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 00:23:33.624079 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:33.623683 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" podUID="f9c6e0c4-cc76-4dff-8073-8c399e461426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 00:23:34.623304 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:34.623275 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 00:23:34.623472 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:34.623326 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" podUID="f9c6e0c4-cc76-4dff-8073-8c399e461426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 00:23:35.794348 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:35.794317 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:35.795352 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:35.795305 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:35.796339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:35.796317 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:23:37.311163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:37.311131 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:39.923299 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:39.923263 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:23:42.049110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:42.049071 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:47.003657 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:47.003617 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:51.818428 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:51.818386 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:52.483412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.483381 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj"] Nov 03 00:23:52.487568 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.487548 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.563260 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.563231 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g79pm\" (UniqueName: \"kubernetes.io/projected/1c78610a-8df0-4f6d-b980-cb697136ab3f-kube-api-access-g79pm\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.563466 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.563277 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-tmp-dir\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.563466 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.563376 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c78610a-8df0-4f6d-b980-cb697136ab3f-tls-certs\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.563466 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.563419 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-model-cache\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.563583 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.563502 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-dshm\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.563583 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.563541 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-home\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.589946 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.589894 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e28e1a\"/\"distinct-2-w1-e28e1a-kserve-self-signed-certs\"" Nov 03 00:23:52.591584 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.591561 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj"] Nov 03 00:23:52.659009 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.658975 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:52.670212 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.670175 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g79pm\" (UniqueName: \"kubernetes.io/projected/1c78610a-8df0-4f6d-b980-cb697136ab3f-kube-api-access-g79pm\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.670470 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.670447 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-tmp-dir\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.670627 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.670610 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c78610a-8df0-4f6d-b980-cb697136ab3f-tls-certs\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.670768 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.670753 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-model-cache\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.670914 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.670898 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-dshm\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.671048 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.671033 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-home\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.671939 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.671915 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-home\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.677078 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.677053 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-tmp-dir\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.678054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.678036 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-model-cache\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.686447 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.686389 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-dshm\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.689348 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.689320 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c78610a-8df0-4f6d-b980-cb697136ab3f-tls-certs\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.734131 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.734058 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g79pm\" (UniqueName: \"kubernetes.io/projected/1c78610a-8df0-4f6d-b980-cb697136ab3f-kube-api-access-g79pm\") pod \"distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:52.800340 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:52.800308 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:53.196246 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:53.196224 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj"] Nov 03 00:23:53.198252 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:23:53.198225 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c78610a_8df0_4f6d_b980_cb697136ab3f.slice/crio-ed334df6c3b061cc88f43be9f61a46b530f5ad6d142a8516e5909bccc368c57c WatchSource:0}: Error finding container ed334df6c3b061cc88f43be9f61a46b530f5ad6d142a8516e5909bccc368c57c: Status 404 returned error can't find the container with id ed334df6c3b061cc88f43be9f61a46b530f5ad6d142a8516e5909bccc368c57c Nov 03 00:23:53.661757 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:53.661720 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" event={"ID":"1c78610a-8df0-4f6d-b980-cb697136ab3f","Type":"ContainerStarted","Data":"ffe6a5150571234eaa49c9d4a067cf7cfc925330a78837a0f57985f048e2dbff"} Nov 03 00:23:53.661973 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:53.661767 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" event={"ID":"1c78610a-8df0-4f6d-b980-cb697136ab3f","Type":"ContainerStarted","Data":"ed334df6c3b061cc88f43be9f61a46b530f5ad6d142a8516e5909bccc368c57c"} Nov 03 00:23:53.777338 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:53.777287 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" podStartSLOduration=1.777272494 podStartE2EDuration="1.777272494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:53.775617781 +0000 UTC m=+1080.726557937" watchObservedRunningTime="2026-11-03 00:23:53.777272494 +0000 UTC m=+1080.728212651" Nov 03 00:23:53.801300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:53.801266 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:23:53.943009 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:53.942910 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc"] Nov 03 00:23:53.948270 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:53.948244 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.002116 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.002084 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ebeeb9-openshift-default-dockercfg-pt5k5\"" Nov 03 00:23:54.006237 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.006209 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc"] Nov 03 00:23:54.020521 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.020479 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.020521 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.020522 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.020732 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.020641 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.020732 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.020692 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bnfg\" (UniqueName: \"kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-kube-api-access-2bnfg\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.020856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.020800 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57107b1c-15e3-448e-9dee-de481f4d4bca-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.020933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.020917 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.020988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.020951 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.021074 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.021058 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.021132 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.021108 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.122234 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.122202 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.122234 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.122232 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.122444 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.122269 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.122444 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.122308 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bnfg\" (UniqueName: \"kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-kube-api-access-2bnfg\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.122444 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.122339 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57107b1c-15e3-448e-9dee-de481f4d4bca-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.122444 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.122376 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.122444 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.122409 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.122680 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.122514 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.122680 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.122562 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.122680 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.122628 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.122870 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.122684 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.122870 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.122802 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.122955 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.122888 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.123063 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.123043 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57107b1c-15e3-448e-9dee-de481f4d4bca-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.125050 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.125030 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.125505 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.125487 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.184209 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.184175 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bnfg\" (UniqueName: \"kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-kube-api-access-2bnfg\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.196181 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.187711 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.262247 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.262212 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:54.704235 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.704201 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc"] Nov 03 00:23:54.745414 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:23:54.745376 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57107b1c_15e3_448e_9dee_de481f4d4bca.slice/crio-fab073acaca6304b5efcaf4afe5cbc4f5594d12c24008dd9256d6528ec67a0a4 WatchSource:0}: Error finding container fab073acaca6304b5efcaf4afe5cbc4f5594d12c24008dd9256d6528ec67a0a4: Status 404 returned error can't find the container with id fab073acaca6304b5efcaf4afe5cbc4f5594d12c24008dd9256d6528ec67a0a4 Nov 03 00:23:54.748041 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.748001 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:23:54.748134 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.748115 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:23:54.748179 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:54.748163 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:23:55.576621 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.576571 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr"] Nov 03 00:23:55.600860 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.595258 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.636413 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.636381 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr"] Nov 03 00:23:55.643163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.642572 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-tmp-dir\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.643163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.642627 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-dshm\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.643163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.642698 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5bdj\" (UniqueName: \"kubernetes.io/projected/f60672ea-e415-4096-bf4f-4689d4194dbf-kube-api-access-j5bdj\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.643163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.642793 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-home\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.643163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.642842 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-model-cache\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.643163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.642895 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f60672ea-e415-4096-bf4f-4689d4194dbf-tls-certs\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.655254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.655222 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"distinct-w3-d7bfc2-kserve-self-signed-certs\"" Nov 03 00:23:55.675622 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.675592 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" event={"ID":"57107b1c-15e3-448e-9dee-de481f4d4bca","Type":"ContainerStarted","Data":"64ab45fab60f0318861a1f5ac72780be7598778e51d6299c5738b76a51d86b3a"} Nov 03 00:23:55.681127 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.681065 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" event={"ID":"57107b1c-15e3-448e-9dee-de481f4d4bca","Type":"ContainerStarted","Data":"fab073acaca6304b5efcaf4afe5cbc4f5594d12c24008dd9256d6528ec67a0a4"} Nov 03 00:23:55.684744 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.684717 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"kube-root-ca.crt\"" Nov 03 00:23:55.685141 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.685123 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"default-dockercfg-mpmcw\"" Nov 03 00:23:55.685330 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.685315 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"openshift-service-ca.crt\"" Nov 03 00:23:55.690204 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.690179 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:55.755561 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.755521 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-home\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.755887 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.755865 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-model-cache\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.756068 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.756054 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f60672ea-e415-4096-bf4f-4689d4194dbf-tls-certs\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.756306 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.756290 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-tmp-dir\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.756452 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.756430 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-dshm\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.756642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.756618 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5bdj\" (UniqueName: \"kubernetes.io/projected/f60672ea-e415-4096-bf4f-4689d4194dbf-kube-api-access-j5bdj\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.757908 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.757885 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-home\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.758716 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.758694 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-model-cache\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.782853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.765364 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-tmp-dir\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.787650 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.786337 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f60672ea-e415-4096-bf4f-4689d4194dbf-tls-certs\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.816857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.810507 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-dshm\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.837898 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.834737 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" podStartSLOduration=2.834715041 podStartE2EDuration="2.834715041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:55.831661526 +0000 UTC m=+1082.782601685" watchObservedRunningTime="2026-11-03 00:23:55.834715041 +0000 UTC m=+1082.785655200" Nov 03 00:23:55.845549 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.845517 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5bdj\" (UniqueName: \"kubernetes.io/projected/f60672ea-e415-4096-bf4f-4689d4194dbf-kube-api-access-j5bdj\") pod \"distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:55.921765 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:55.921727 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:56.263243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:56.263213 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:23:56.264311 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:56.264290 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:23:56.264379 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:56.264332 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" podUID="57107b1c-15e3-448e-9dee-de481f4d4bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 00:23:56.501850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:56.498158 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr"] Nov 03 00:23:56.713199 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:56.711916 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" event={"ID":"f60672ea-e415-4096-bf4f-4689d4194dbf","Type":"ContainerStarted","Data":"4bb68f545c59da36c217afa86f128b14d7fa1f96d1cbed815d92582863c424af"} Nov 03 00:23:56.713199 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:56.711970 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" event={"ID":"f60672ea-e415-4096-bf4f-4689d4194dbf","Type":"ContainerStarted","Data":"d4e1bcb36c3afb8a2b68f75b2e673055b092b5f5e05d3fee88358911839ac8a9"} Nov 03 00:23:56.713746 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:56.713234 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:56.828006 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:56.827490 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" podStartSLOduration=1.827474488 podStartE2EDuration="1.827474488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:56.825647307 +0000 UTC m=+1083.776587465" watchObservedRunningTime="2026-11-03 00:23:56.827474488 +0000 UTC m=+1083.778414642" Nov 03 00:23:56.923082 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:56.923048 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:23:57.154271 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:23:57.154163 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:23:57.263385 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:57.263349 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:23:57.263552 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:57.263406 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" podUID="57107b1c-15e3-448e-9dee-de481f4d4bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 00:23:58.263273 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:58.263233 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:23:58.263837 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:58.263290 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" podUID="57107b1c-15e3-448e-9dee-de481f4d4bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 00:23:59.265303 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:59.265263 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:23:59.265956 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:23:59.265911 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" podUID="57107b1c-15e3-448e-9dee-de481f4d4bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 00:24:00.263099 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:00.263060 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:24:00.263409 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:00.263376 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" podUID="57107b1c-15e3-448e-9dee-de481f4d4bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 00:24:01.263291 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:01.263212 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:24:01.263291 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:01.263276 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" podUID="57107b1c-15e3-448e-9dee-de481f4d4bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 00:24:01.630612 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:01.630571 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:02.267746 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:02.267716 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:24:02.268215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:02.268031 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:24:02.268930 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:02.268916 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:24:02.801357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:02.801318 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:24:02.814450 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:02.814424 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:24:03.778874 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:03.778848 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:24:04.315315 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:04.315281 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:05.922805 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:05.922766 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:24:05.934966 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:05.934938 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:24:06.287863 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:06.287831 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:06.795091 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:06.795062 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 00:24:08.432618 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.432588 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5"] Nov 03 00:24:08.436360 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.436340 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.488227 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.488200 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-048135-openshift-default-dockercfg-xt942\"" Nov 03 00:24:08.540655 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.540623 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5"] Nov 03 00:24:08.583642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.583605 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltt2m\" (UniqueName: \"kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-kube-api-access-ltt2m\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.583842 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.583659 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.583842 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.583768 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.583842 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.583799 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.583842 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.583839 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.583989 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.583904 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23271791-3a3a-4695-aa8d-33362c295435-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.583989 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.583964 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.584070 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.584057 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.584103 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.584088 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23271791-3a3a-4695-aa8d-33362c295435-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.685163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.685076 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23271791-3a3a-4695-aa8d-33362c295435-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.685326 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.685176 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltt2m\" (UniqueName: \"kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-kube-api-access-ltt2m\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.685326 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.685208 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.685326 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.685237 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.685326 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.685269 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.685326 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.685297 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.685326 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.685326 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23271791-3a3a-4695-aa8d-33362c295435-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.685652 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.685384 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.685652 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.685468 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.685802 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.685776 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.685891 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.685802 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.686197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.686087 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.687839 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.686140 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.687839 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.686629 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23271791-3a3a-4695-aa8d-33362c295435-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.688791 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.688750 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23271791-3a3a-4695-aa8d-33362c295435-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.695685 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.695662 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.741617 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.741588 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltt2m\" (UniqueName: \"kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-kube-api-access-ltt2m\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.741617 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.741600 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-jdwq5\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:08.749923 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:08.749888 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:09.179521 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:24:09.179490 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23271791_3a3a_4695_aa8d_33362c295435.slice/crio-f40b8202c4308dc938e8d2d5352ebcbbe17e3440ba92296c3332b144a6829d22 WatchSource:0}: Error finding container f40b8202c4308dc938e8d2d5352ebcbbe17e3440ba92296c3332b144a6829d22: Status 404 returned error can't find the container with id f40b8202c4308dc938e8d2d5352ebcbbe17e3440ba92296c3332b144a6829d22 Nov 03 00:24:09.179992 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:09.179972 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5"] Nov 03 00:24:09.181991 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:09.181957 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:24:09.182078 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:09.182021 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:24:09.182078 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:09.182049 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:24:09.801848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:09.801781 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" event={"ID":"23271791-3a3a-4695-aa8d-33362c295435","Type":"ContainerStarted","Data":"9260f57af9fd3e2c1308eb0d20f6c8abdb87fdb4e4354e1e1d5c2aaa7d81fd98"} Nov 03 00:24:09.801848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:09.801855 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" event={"ID":"23271791-3a3a-4695-aa8d-33362c295435","Type":"ContainerStarted","Data":"f40b8202c4308dc938e8d2d5352ebcbbe17e3440ba92296c3332b144a6829d22"} Nov 03 00:24:09.939494 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:09.939439 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" podStartSLOduration=1.939425517 podStartE2EDuration="1.939425517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:09.936861376 +0000 UTC m=+1096.887801535" watchObservedRunningTime="2026-11-03 00:24:09.939425517 +0000 UTC m=+1096.890366079" Nov 03 00:24:10.750423 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:10.750392 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:10.751971 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:10.751944 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:24:10.752102 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:10.751995 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" podUID="23271791-3a3a-4695-aa8d-33362c295435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 00:24:11.145843 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:11.145782 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:11.751136 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:11.751097 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:24:11.751343 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:11.751163 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" podUID="23271791-3a3a-4695-aa8d-33362c295435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 00:24:11.963712 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:11.963674 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:24:12.751231 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:12.751194 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:24:12.751733 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:12.751244 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" podUID="23271791-3a3a-4695-aa8d-33362c295435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 00:24:13.758172 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:13.758131 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:24:13.758912 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:13.758874 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" podUID="23271791-3a3a-4695-aa8d-33362c295435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 00:24:14.751122 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:14.751084 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:24:14.751311 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:14.751149 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" podUID="23271791-3a3a-4695-aa8d-33362c295435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 00:24:15.750359 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:15.750324 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:24:15.750782 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:15.750380 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" podUID="23271791-3a3a-4695-aa8d-33362c295435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 00:24:16.261543 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:16.261506 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:17.791007 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:17.790958 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:24:17.791544 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:17.791023 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" podUID="23271791-3a3a-4695-aa8d-33362c295435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:18.118354 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:18.118268 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:18.118674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:18.118646 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:18.119624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:18.119602 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:24:21.018499 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:21.018459 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:25.657847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.655184 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5"] Nov 03 00:24:25.675219 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.675189 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.723204 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.723166 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5"] Nov 03 00:24:25.725059 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.724882 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:25.769853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.769768 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"distinct-2-w3-d7bfc2-kserve-self-signed-certs\"" Nov 03 00:24:25.778521 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.778488 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tmp-dir\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.778711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.778562 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-dshm\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.778711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.778633 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-model-cache\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.778711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.778656 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tls-certs\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.778711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.778681 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b662r\" (UniqueName: \"kubernetes.io/projected/eaf4988a-ac9f-446e-91ec-28ee63f944a7-kube-api-access-b662r\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.778900 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.778729 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-home\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.880355 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.880320 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-model-cache\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.880355 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.880357 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tls-certs\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.880561 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.880374 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b662r\" (UniqueName: \"kubernetes.io/projected/eaf4988a-ac9f-446e-91ec-28ee63f944a7-kube-api-access-b662r\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.880561 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.880411 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-home\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.880561 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.880483 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tmp-dir\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.880561 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.880521 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-dshm\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.880873 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.880797 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-model-cache\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.881244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.881222 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-home\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.881498 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.881475 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tmp-dir\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.886393 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.886362 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tls-certs\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.886856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.886809 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-dshm\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:25.891930 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.891907 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:25.937485 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:25.937407 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b662r\" (UniqueName: \"kubernetes.io/projected/eaf4988a-ac9f-446e-91ec-28ee63f944a7-kube-api-access-b662r\") pod \"distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:26.001340 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:26.001046 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:26.566080 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:26.566054 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5"] Nov 03 00:24:26.568417 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:24:26.568393 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeaf4988a_ac9f_446e_91ec_28ee63f944a7.slice/crio-e6e23c30d924a9cde90b52ad7ddbbd9c46e7cfc5283dcb06fbcc424c0fd319ea WatchSource:0}: Error finding container e6e23c30d924a9cde90b52ad7ddbbd9c46e7cfc5283dcb06fbcc424c0fd319ea: Status 404 returned error can't find the container with id e6e23c30d924a9cde90b52ad7ddbbd9c46e7cfc5283dcb06fbcc424c0fd319ea Nov 03 00:24:26.911533 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:26.911444 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" event={"ID":"eaf4988a-ac9f-446e-91ec-28ee63f944a7","Type":"ContainerStarted","Data":"03e0dd81c9b62cca3890c26b7f8cf03aa607b5d7b936c6aee7aa6b245fb43050"} Nov 03 00:24:26.911533 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:26.911498 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" event={"ID":"eaf4988a-ac9f-446e-91ec-28ee63f944a7","Type":"ContainerStarted","Data":"e6e23c30d924a9cde90b52ad7ddbbd9c46e7cfc5283dcb06fbcc424c0fd319ea"} Nov 03 00:24:27.002568 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:27.002530 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:27.029019 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:27.028851 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" podStartSLOduration=2.028797853 podStartE2EDuration="2.028797853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:27.028538168 +0000 UTC m=+1113.979478320" watchObservedRunningTime="2026-11-03 00:24:27.028797853 +0000 UTC m=+1113.979738011" Nov 03 00:24:27.226697 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:24:27.226611 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:24:30.810630 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:30.810599 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:35.527941 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:35.527904 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:35.999964 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:35.999934 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg"] Nov 03 00:24:36.004633 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.004609 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:36.004765 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.004710 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.016416 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.016392 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:36.022395 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.022369 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8kz7\" (UniqueName: \"kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-kube-api-access-t8kz7\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.022497 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.022408 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.022497 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.022468 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.022586 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.022574 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3cd93c-0280-450f-9445-eab9644817db-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.022663 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.022601 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.022663 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.022621 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.022761 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.022688 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.022886 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.022867 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.022943 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.022907 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3cd93c-0280-450f-9445-eab9644817db-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.062109 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.062081 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg"] Nov 03 00:24:36.108982 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.108945 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e28e1a-openshift-default-dockercfg-4zwkf\"" Nov 03 00:24:36.123486 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.123461 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3cd93c-0280-450f-9445-eab9644817db-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.123615 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.123498 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.123615 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.123602 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.123726 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.123650 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.123784 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.123725 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.123784 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.123758 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3cd93c-0280-450f-9445-eab9644817db-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.123905 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.123787 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8kz7\" (UniqueName: \"kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-kube-api-access-t8kz7\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.123905 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.123859 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.124023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.123902 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.124023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.123905 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.124242 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.124214 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.124242 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.124234 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.124366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.124317 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.124366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.124333 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3cd93c-0280-450f-9445-eab9644817db-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.126374 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.126350 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.126550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.126529 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3cd93c-0280-450f-9445-eab9644817db-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.179920 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.179895 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.204292 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.204266 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8kz7\" (UniqueName: \"kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-kube-api-access-t8kz7\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.319327 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.319293 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:36.744991 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.744966 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg"] Nov 03 00:24:36.746862 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:24:36.746802 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3cd93c_0280_450f_9445_eab9644817db.slice/crio-7a0a06920eddeb083e83d07f9fb3958572082f10d3d01767e7ba5588a98d5fe8 WatchSource:0}: Error finding container 7a0a06920eddeb083e83d07f9fb3958572082f10d3d01767e7ba5588a98d5fe8: Status 404 returned error can't find the container with id 7a0a06920eddeb083e83d07f9fb3958572082f10d3d01767e7ba5588a98d5fe8 Nov 03 00:24:36.749579 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.749538 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:24:36.749654 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.749629 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:24:36.749689 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.749667 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:24:36.967705 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.967663 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" event={"ID":"4e3cd93c-0280-450f-9445-eab9644817db","Type":"ContainerStarted","Data":"95d7194da63e0b0e770ac9fb74ec484be9d9996823e4a5743defd47cc8c6daff"} Nov 03 00:24:36.967970 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.967725 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" event={"ID":"4e3cd93c-0280-450f-9445-eab9644817db","Type":"ContainerStarted","Data":"7a0a06920eddeb083e83d07f9fb3958572082f10d3d01767e7ba5588a98d5fe8"} Nov 03 00:24:36.979382 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:36.979357 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 00:24:37.084886 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:37.084799 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" podStartSLOduration=2.084783641 podStartE2EDuration="2.084783641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:37.08394653 +0000 UTC m=+1124.034886688" watchObservedRunningTime="2026-11-03 00:24:37.084783641 +0000 UTC m=+1124.035723794" Nov 03 00:24:37.320398 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:37.320359 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:37.322160 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:37.322134 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 00:24:37.322292 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:37.322186 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" podUID="4e3cd93c-0280-450f-9445-eab9644817db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 00:24:38.320038 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.320003 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 00:24:38.320445 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.320059 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" podUID="4e3cd93c-0280-450f-9445-eab9644817db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 00:24:38.702375 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.702274 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl"] Nov 03 00:24:38.706108 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.706082 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.758665 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.758630 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggrh5\" (UniqueName: \"kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-kube-api-access-ggrh5\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.758846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.758681 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7ab390-893f-4d69-982a-464bf5483fa5-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.758846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.758705 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.758846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.758787 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.759036 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.758974 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.759036 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.759016 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.759142 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.759067 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.759200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.759168 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.759255 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.759206 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.807910 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.796798 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl"] Nov 03 00:24:38.860280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.860245 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7ab390-893f-4d69-982a-464bf5483fa5-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.860280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.860287 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.860507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.860314 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.860507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.860393 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.860507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.860422 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.860507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.860443 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.860507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.860491 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.860767 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.860525 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.860767 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.860696 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrh5\" (UniqueName: \"kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-kube-api-access-ggrh5\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.860921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.860898 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.861014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.860984 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.861097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.861014 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.861193 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.861166 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7ab390-893f-4d69-982a-464bf5483fa5-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.861193 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.861192 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.862955 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.862935 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.863074 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.863057 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.917247 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.917218 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:38.917434 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:38.917390 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrh5\" (UniqueName: \"kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-kube-api-access-ggrh5\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-p5rhl\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:39.020269 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.020189 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:39.304522 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.304487 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp"] Nov 03 00:24:39.309515 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.309479 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.320149 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.320124 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 00:24:39.320492 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.320175 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" podUID="4e3cd93c-0280-450f-9445-eab9644817db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 00:24:39.365743 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.365712 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp"] Nov 03 00:24:39.367522 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.367493 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.367674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.367553 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.367674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.367637 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7frv\" (UniqueName: \"kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-kube-api-access-h7frv\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.367792 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.367686 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.367792 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.367733 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.367792 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.367774 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.367970 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.367815 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.367970 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.367933 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85c2951-14fc-4025-99b5-d78d7794cd0c-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.368158 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.368140 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.469147 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.469116 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.469338 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.469179 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.469338 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.469225 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.469338 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.469301 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7frv\" (UniqueName: \"kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-kube-api-access-h7frv\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.469492 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.469348 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.469492 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.469379 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.469492 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.469422 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.469492 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.469464 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.469690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.469498 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85c2951-14fc-4025-99b5-d78d7794cd0c-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.469690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.469595 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.469801 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.469774 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.469872 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.469805 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.469923 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.469867 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.470163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.470144 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85c2951-14fc-4025-99b5-d78d7794cd0c-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.471742 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.471714 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.472114 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.472095 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.523849 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.523805 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl"] Nov 03 00:24:39.525957 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:24:39.525911 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7ab390_893f_4d69_982a_464bf5483fa5.slice/crio-8d683e0c9a70f03ed07e1486bdc1c2b932abfd118ee1d7fc0787ba3f06ffe6cc WatchSource:0}: Error finding container 8d683e0c9a70f03ed07e1486bdc1c2b932abfd118ee1d7fc0787ba3f06ffe6cc: Status 404 returned error can't find the container with id 8d683e0c9a70f03ed07e1486bdc1c2b932abfd118ee1d7fc0787ba3f06ffe6cc Nov 03 00:24:39.526517 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.526469 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.527022 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.527001 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7frv\" (UniqueName: \"kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-kube-api-access-h7frv\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:39.528378 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.528345 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:24:39.528433 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.528421 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:24:39.528468 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.528447 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:24:39.624003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:39.623973 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:40.010336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:40.010241 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" event={"ID":"4e7ab390-893f-4d69-982a-464bf5483fa5","Type":"ContainerStarted","Data":"be9998f569960e28d32cb0ae4093f845f3fa61e43e931aeb2b5329756307e93f"} Nov 03 00:24:40.010336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:40.010301 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" event={"ID":"4e7ab390-893f-4d69-982a-464bf5483fa5","Type":"ContainerStarted","Data":"8d683e0c9a70f03ed07e1486bdc1c2b932abfd118ee1d7fc0787ba3f06ffe6cc"} Nov 03 00:24:40.020615 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:40.020588 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:40.022542 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:40.022518 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 00:24:40.022659 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:40.022583 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" podUID="4e7ab390-893f-4d69-982a-464bf5483fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 00:24:40.104853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:40.095615 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp"] Nov 03 00:24:40.110738 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:24:40.110685 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85c2951_14fc_4025_99b5_d78d7794cd0c.slice/crio-3edf0e6b388e95106f7490e99d0585e89f5f4f31def170ff67188de97c9d8aea WatchSource:0}: Error finding container 3edf0e6b388e95106f7490e99d0585e89f5f4f31def170ff67188de97c9d8aea: Status 404 returned error can't find the container with id 3edf0e6b388e95106f7490e99d0585e89f5f4f31def170ff67188de97c9d8aea Nov 03 00:24:40.113896 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:40.113855 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:24:40.113994 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:40.113970 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:24:40.114031 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:40.114017 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:24:40.131163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:40.131119 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" podStartSLOduration=2.131107085 podStartE2EDuration="2.131107085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:40.129810842 +0000 UTC m=+1127.080751000" watchObservedRunningTime="2026-11-03 00:24:40.131107085 +0000 UTC m=+1127.082047242" Nov 03 00:24:40.330362 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:40.324243 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 00:24:40.330362 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:40.324307 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" podUID="4e3cd93c-0280-450f-9445-eab9644817db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 00:24:40.339757 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:40.339690 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:41.036879 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:41.034008 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 00:24:41.036879 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:41.034076 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" podUID="4e7ab390-893f-4d69-982a-464bf5483fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 00:24:41.043591 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:41.043470 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" event={"ID":"d85c2951-14fc-4025-99b5-d78d7794cd0c","Type":"ContainerStarted","Data":"69e6c156d02e4c0fabfb7ed84ec0cbca50e9efd11558419e874998a4288f0fc8"} Nov 03 00:24:41.043591 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:41.043540 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" event={"ID":"d85c2951-14fc-4025-99b5-d78d7794cd0c","Type":"ContainerStarted","Data":"3edf0e6b388e95106f7490e99d0585e89f5f4f31def170ff67188de97c9d8aea"} Nov 03 00:24:41.163934 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:41.163860 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" podStartSLOduration=2.163803298 podStartE2EDuration="2.163803298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:41.161619379 +0000 UTC m=+1128.112559532" watchObservedRunningTime="2026-11-03 00:24:41.163803298 +0000 UTC m=+1128.114743456" Nov 03 00:24:41.320387 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:41.320285 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 00:24:41.323951 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:41.323900 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" podUID="4e3cd93c-0280-450f-9445-eab9644817db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 00:24:41.631948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:41.625988 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:41.631948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:41.627325 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 00:24:41.631948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:41.627380 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" podUID="d85c2951-14fc-4025-99b5-d78d7794cd0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 00:24:42.023379 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:42.023275 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 00:24:42.023842 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:42.023651 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" podUID="4e7ab390-893f-4d69-982a-464bf5483fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 00:24:42.325849 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:42.324265 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 00:24:42.325849 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:42.324332 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" podUID="4e3cd93c-0280-450f-9445-eab9644817db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 00:24:42.632915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:42.631241 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 00:24:42.632915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:42.631312 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" podUID="d85c2951-14fc-4025-99b5-d78d7794cd0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 00:24:43.023322 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:43.023223 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 00:24:43.023634 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:43.023593 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" podUID="4e7ab390-893f-4d69-982a-464bf5483fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 00:24:43.337337 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:43.337296 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 00:24:43.337607 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:43.337573 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" podUID="4e3cd93c-0280-450f-9445-eab9644817db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 00:24:43.630507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:43.630331 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 00:24:43.632029 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:43.630681 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" podUID="d85c2951-14fc-4025-99b5-d78d7794cd0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 00:24:44.028913 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:44.024229 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 00:24:44.028913 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:44.024292 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" podUID="4e7ab390-893f-4d69-982a-464bf5483fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 00:24:44.632278 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:44.632236 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 00:24:44.632573 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:44.632542 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" podUID="d85c2951-14fc-4025-99b5-d78d7794cd0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 00:24:44.674274 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:44.674215 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:44.675206 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:44.675185 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:44.683849 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:44.676023 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:24:45.030349 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:45.030246 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 00:24:45.030349 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:45.030309 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" podUID="4e7ab390-893f-4d69-982a-464bf5483fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 00:24:45.545904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:45.545875 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:45.636852 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:45.629606 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 00:24:45.636852 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:45.629672 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" podUID="d85c2951-14fc-4025-99b5-d78d7794cd0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 00:24:46.020559 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:46.020465 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 00:24:46.020559 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:46.020532 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" podUID="4e7ab390-893f-4d69-982a-464bf5483fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 00:24:46.625416 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:46.625374 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 00:24:46.625861 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:46.625446 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" podUID="d85c2951-14fc-4025-99b5-d78d7794cd0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 00:24:47.184005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:47.183974 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:47.184280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:47.184252 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:47.185363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:47.185344 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:24:47.880242 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:47.880213 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:47.880673 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:47.880560 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:47.881675 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:47.881649 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:24:49.323289 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:49.323255 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:24:50.743847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:50.743088 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:51.769852 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:51.769192 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" podUID="89d21cbe-9619-4d67-bf52-105653fa5e73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:51.791847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:51.782907 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" podUID="1c78610a-8df0-4f6d-b980-cb697136ab3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:51.843161 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:51.842074 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:24:51.843161 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:51.842159 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" podUID="f9c6e0c4-cc76-4dff-8073-8c399e461426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.360853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:52.359926 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" podUID="ca16bbd6-e0a8-4d24-ba33-a749b8659aee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.510848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:52.510127 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" podUID="1f7d333a-0851-4b86-a3ab-29b6ea453fa7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.803848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:52.794992 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" podUID="f60672ea-e415-4096-bf4f-4689d4194dbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.803848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:52.795368 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" podUID="1c78610a-8df0-4f6d-b980-cb697136ab3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 00:24:52.803848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:52.795947 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:24:52.803848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:52.800852 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" podUID="89d21cbe-9619-4d67-bf52-105653fa5e73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.803848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:52.801234 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:24:52.803848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:52.802355 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" podUID="2fd1b7e8-1357-47d7-9ba6-236e95443aae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.970847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:52.969851 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" podUID="eaf4988a-ac9f-446e-91ec-28ee63f944a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:53.028893 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:53.028857 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 00:24:53.041286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:53.041208 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 00:24:55.818486 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:24:55.818434 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:57.155934 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:24:57.155891 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:25:00.513849 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:00.513787 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:04.654999 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:04.654953 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:05.422382 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:05.422345 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:07.230706 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:07.230672 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:07.820405 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:07.805202 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:08.140499 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:08.140403 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:08.302234 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:08.302195 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:08.923831 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:08.923783 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc"] Nov 03 00:25:08.928036 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:08.928018 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.032303 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.032265 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.032513 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.032391 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.032592 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.032537 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.032648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.032617 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.032790 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.032762 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699aded9-a381-4ee0-820e-3c568b5a4ad7-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.032911 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.032802 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.032973 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.032916 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s48l\" (UniqueName: \"kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-kube-api-access-2s48l\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.033036 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.032994 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.033036 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.033026 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.034492 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.034467 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc"] Nov 03 00:25:09.134360 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.134329 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.134360 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.134361 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.134640 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.134528 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.134640 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.134575 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.134751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.134647 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.134751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.134697 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.134751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.134750 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699aded9-a381-4ee0-820e-3c568b5a4ad7-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.134949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.134773 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.134949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.134792 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s48l\" (UniqueName: \"kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-kube-api-access-2s48l\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.135197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.135172 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.135301 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.135277 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.135454 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.135426 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.135578 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.135477 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.135794 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.135776 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699aded9-a381-4ee0-820e-3c568b5a4ad7-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.137523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.137502 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.137626 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.137593 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.191747 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.191665 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.191924 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.191843 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s48l\" (UniqueName: \"kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-kube-api-access-2s48l\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-wzwgc\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.241636 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.241605 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:09.528521 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.528429 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp"] Nov 03 00:25:09.534272 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.534246 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.592626 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.592588 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp"] Nov 03 00:25:09.642047 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.642014 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.642047 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.642049 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nlj2\" (UniqueName: \"kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-kube-api-access-4nlj2\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.642252 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.642149 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.642252 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.642222 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.642252 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.642239 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.642366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.642284 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.642366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.642347 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.642456 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.642429 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.642585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.642566 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.695816 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.695793 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc"] Nov 03 00:25:09.697773 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:25:09.697744 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699aded9_a381_4ee0_820e_3c568b5a4ad7.slice/crio-8b6674402970b0a27db7e427aac900cf27b91e6016ce59bbb7717290770ddcaf WatchSource:0}: Error finding container 8b6674402970b0a27db7e427aac900cf27b91e6016ce59bbb7717290770ddcaf: Status 404 returned error can't find the container with id 8b6674402970b0a27db7e427aac900cf27b91e6016ce59bbb7717290770ddcaf Nov 03 00:25:09.700269 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.700231 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:25:09.700359 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.700300 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:25:09.700359 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.700351 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:25:09.744161 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.744134 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.744284 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.744224 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.744284 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.744261 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.744388 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.744281 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nlj2\" (UniqueName: \"kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-kube-api-access-4nlj2\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.744388 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.744325 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.744388 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.744368 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.744530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.744391 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.744530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.744441 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.744530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.744468 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.744891 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.744863 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.745048 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.744908 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.745114 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.744958 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.745155 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.745140 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.745329 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.745310 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.746691 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.746675 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.746976 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.746955 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.800530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.800501 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.800705 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.800545 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nlj2\" (UniqueName: \"kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-kube-api-access-4nlj2\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:09.850501 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:09.850465 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:10.115237 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.115147 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:10.277331 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.277305 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp"] Nov 03 00:25:10.279287 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:25:10.279255 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f7ad4c_9382_4007_96ea_06ed4b82a7ca.slice/crio-faf0198f874eae91837b567a8f7d54e06d2743179e2cda57d0b1aee032ae69ed WatchSource:0}: Error finding container faf0198f874eae91837b567a8f7d54e06d2743179e2cda57d0b1aee032ae69ed: Status 404 returned error can't find the container with id faf0198f874eae91837b567a8f7d54e06d2743179e2cda57d0b1aee032ae69ed Nov 03 00:25:10.281771 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.281738 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:25:10.281860 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.281804 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:25:10.281964 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.281871 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:25:10.414956 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.414842 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" event={"ID":"46f7ad4c-9382-4007-96ea-06ed4b82a7ca","Type":"ContainerStarted","Data":"b1a09acd73a9ecdb949bb69282d1fd01f2ad2a2ca37dc4a075dd2886a7b5a70d"} Nov 03 00:25:10.414956 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.414949 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" event={"ID":"46f7ad4c-9382-4007-96ea-06ed4b82a7ca","Type":"ContainerStarted","Data":"faf0198f874eae91837b567a8f7d54e06d2743179e2cda57d0b1aee032ae69ed"} Nov 03 00:25:10.417861 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.417785 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" event={"ID":"699aded9-a381-4ee0-820e-3c568b5a4ad7","Type":"ContainerStarted","Data":"c1a390a4b784185a0a924fb7e767011467c274e606ab4542e1344e8c698ec79b"} Nov 03 00:25:10.418035 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.417865 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" event={"ID":"699aded9-a381-4ee0-820e-3c568b5a4ad7","Type":"ContainerStarted","Data":"8b6674402970b0a27db7e427aac900cf27b91e6016ce59bbb7717290770ddcaf"} Nov 03 00:25:10.450715 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.450681 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:10.535587 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.535523 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" podStartSLOduration=1.535502393 podStartE2EDuration="1.535502393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:10.533199188 +0000 UTC m=+1157.484139347" watchObservedRunningTime="2026-11-03 00:25:10.535502393 +0000 UTC m=+1157.486442551" Nov 03 00:25:10.614678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.614634 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:10.657198 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.656623 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" podStartSLOduration=2.656602167 podStartE2EDuration="2.656602167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:10.654277485 +0000 UTC m=+1157.605217643" watchObservedRunningTime="2026-11-03 00:25:10.656602167 +0000 UTC m=+1157.607542325" Nov 03 00:25:10.851594 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:10.851557 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:11.242520 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:11.242423 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:11.245188 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:11.243877 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:25:11.245188 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:11.243929 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" podUID="699aded9-a381-4ee0-820e-3c568b5a4ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 00:25:11.853350 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:11.853313 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:25:11.853754 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:11.853366 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" podUID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 00:25:12.137041 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:12.136945 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:12.242948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:12.242912 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:25:12.243148 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:12.242978 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" podUID="699aded9-a381-4ee0-820e-3c568b5a4ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 00:25:12.359953 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:12.359922 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:12.851322 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:12.851286 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:25:12.851514 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:12.851332 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" podUID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 00:25:13.243003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.242899 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:25:13.243003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.242973 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" podUID="699aded9-a381-4ee0-820e-3c568b5a4ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 00:25:13.613907 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.613858 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8"] Nov 03 00:25:13.623676 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.623648 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.657181 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.657146 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.657325 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.657235 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.657325 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.657264 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b261c014-5911-4da3-8ba9-6ed91a665c0d-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.657325 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.657291 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2599l\" (UniqueName: \"kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-kube-api-access-2599l\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.657325 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.657320 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.657485 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.657344 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.657485 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.657372 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.657664 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.657628 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.657765 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.657751 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.759257 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.759220 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.759428 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.759308 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.759428 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.759369 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.759428 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.759405 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.759428 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.759423 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b261c014-5911-4da3-8ba9-6ed91a665c0d-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.759562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.759445 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2599l\" (UniqueName: \"kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-kube-api-access-2599l\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.759562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.759463 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.759562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.759477 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.759562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.759494 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.760541 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.760515 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.761239 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.761208 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b261c014-5911-4da3-8ba9-6ed91a665c0d-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.761340 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.760972 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.761340 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.761187 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.761340 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.760751 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.762554 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.762536 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.767216 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.767142 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.839885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.839180 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.855881 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.851142 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:25:13.862224 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.862181 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" podUID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 00:25:13.862860 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.857616 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2599l\" (UniqueName: \"kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-kube-api-access-2599l\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:13.865317 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.865251 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8"] Nov 03 00:25:13.948513 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:13.948475 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:14.257965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:14.246329 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:25:14.257965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:14.246402 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" podUID="699aded9-a381-4ee0-820e-3c568b5a4ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 00:25:14.837356 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:14.837310 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8"] Nov 03 00:25:14.860883 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:14.860716 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:25:14.860883 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:14.860769 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" podUID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 00:25:14.862317 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:14.862277 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:25:14.862523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:14.862498 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:25:14.862674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:14.862654 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:25:15.247859 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:15.246949 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:25:15.247859 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:15.247013 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" podUID="699aded9-a381-4ee0-820e-3c568b5a4ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 00:25:15.504921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:15.500779 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" event={"ID":"b261c014-5911-4da3-8ba9-6ed91a665c0d","Type":"ContainerStarted","Data":"79e7cc57ce89f39246a1c05ff70b3c88071fabc6bbab82e9a1a10796aa21703d"} Nov 03 00:25:15.504921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:15.500873 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" event={"ID":"b261c014-5911-4da3-8ba9-6ed91a665c0d","Type":"ContainerStarted","Data":"06dbdb51caac7469f308b77ad90ceb7b3c763d8e2414fbe22f3095846263f3dd"} Nov 03 00:25:15.639961 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:15.637576 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" podStartSLOduration=2.637553181 podStartE2EDuration="2.637553181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:15.628600987 +0000 UTC m=+1162.579541145" watchObservedRunningTime="2026-11-03 00:25:15.637553181 +0000 UTC m=+1162.588493341" Nov 03 00:25:15.853859 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:15.853351 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:25:15.853859 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:15.853434 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" podUID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 00:25:15.953848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:15.952985 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:16.018028 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:16.017985 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" podUID="eaf4988a-ac9f-446e-91ec-28ee63f944a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:16.058120 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:16.058067 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" podUID="1c78610a-8df0-4f6d-b980-cb697136ab3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:16.102419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:16.102002 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" podUID="89d21cbe-9619-4d67-bf52-105653fa5e73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:16.179882 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:16.179768 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" podUID="50ce1488-84fe-4013-970a-83af8bea40b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 03 00:25:16.194140 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:16.193957 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:16.280600 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:16.280485 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:25:16.280600 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:16.280542 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" podUID="699aded9-a381-4ee0-820e-3c568b5a4ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 00:25:16.439789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:16.439682 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" podUID="ca16bbd6-e0a8-4d24-ba33-a749b8659aee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:16.539846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:16.535469 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:16.718896 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:16.718792 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:16.920045 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:16.919989 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:25:16.941291 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:16.941243 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" podUID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 00:25:17.024205 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:17.024006 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 00:25:17.032099 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:17.031907 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" podUID="b261c014-5911-4da3-8ba9-6ed91a665c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 00:25:17.242229 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:17.242196 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 00:25:17.242463 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:17.242434 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" podUID="699aded9-a381-4ee0-820e-3c568b5a4ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 00:25:17.851506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:17.851469 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 00:25:17.852072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:17.851525 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" podUID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 00:25:17.950196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:17.950153 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 00:25:17.950381 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:17.950222 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" podUID="b261c014-5911-4da3-8ba9-6ed91a665c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 00:25:18.403848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:18.403262 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:18.406074 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:18.404889 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:18.414180 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:18.414155 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:25:18.949320 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:18.949282 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 00:25:18.951112 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:18.951071 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" podUID="b261c014-5911-4da3-8ba9-6ed91a665c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 00:25:19.901032 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:19.900979 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:25:19.901394 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:19.901359 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" podUID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:19.950082 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:19.950036 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 00:25:19.950493 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:19.950110 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" podUID="b261c014-5911-4da3-8ba9-6ed91a665c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 00:25:20.125353 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:20.125318 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:20.125890 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:20.125808 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:20.127269 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:20.127247 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:25:20.949256 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:20.949212 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 00:25:20.949447 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:20.949276 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" podUID="b261c014-5911-4da3-8ba9-6ed91a665c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 00:25:21.402601 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:21.402556 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:21.653949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:21.653844 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:21.863437 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:21.863402 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:21.949691 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:21.949604 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 00:25:21.949691 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:21.949662 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" podUID="b261c014-5911-4da3-8ba9-6ed91a665c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 00:25:22.949704 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:22.949667 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 00:25:22.950124 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:22.949723 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" podUID="b261c014-5911-4da3-8ba9-6ed91a665c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 00:25:23.954364 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:23.954333 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:23.955354 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:23.955328 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:23.956244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:23.956228 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:25:26.441288 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:26.441255 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:27.152582 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:25:27.152550 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:25:31.174302 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:31.174267 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:31.533436 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:31.533402 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:32.201120 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:32.200971 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:32.378988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:32.378956 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:34.783702 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:34.783666 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:34.948499 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:34.948465 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:35.096958 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:35.096869 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:35.904992 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:35.904955 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:36.144150 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:36.144116 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:36.430019 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:36.429982 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:38.968911 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:38.968879 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:25:39.435148 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:39.435112 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:40.706850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:40.705266 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:41.110849 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:41.110512 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:41.988866 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:41.971928 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" podUID="eaf4988a-ac9f-446e-91ec-28ee63f944a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:42.043850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:42.041928 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" podUID="1c78610a-8df0-4f6d-b980-cb697136ab3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:42.074850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:42.074008 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" podUID="89d21cbe-9619-4d67-bf52-105653fa5e73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:42.445845 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:42.409234 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" podUID="ca16bbd6-e0a8-4d24-ba33-a749b8659aee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:42.486387 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:42.486334 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:43.019199 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:43.019163 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:44.366777 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:44.366723 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:44.593726 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:44.593165 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:46.299316 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:46.299271 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:46.486875 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:46.486808 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:46.658139 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:46.658048 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:47.805502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:47.805463 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:48.134239 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:48.134158 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:48.303758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:48.303724 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:48.883284 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:48.883248 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:49.059676 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:49.059622 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:49.516147 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:49.516111 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:50.534562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:50.534527 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:53.222216 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:53.222185 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:54.542865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:54.542810 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:54.726336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:54.726283 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:57.154189 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:25:57.154147 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:25:58.031418 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:25:58.031384 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:02.854408 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:02.854375 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:05.188998 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:05.188957 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:07.612171 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:07.612138 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:12.052203 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.052172 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs"] Nov 03 00:26:12.056023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.056006 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.109584 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.109545 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a9404c-serving-cert\"" Nov 03 00:26:12.112890 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.112866 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs"] Nov 03 00:26:12.236380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.236346 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xfjr\" (UniqueName: \"kubernetes.io/projected/a44bddcd-a44c-4998-8c3e-4a91db85655f-kube-api-access-4xfjr\") pod \"maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.236525 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.236424 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44bddcd-a44c-4998-8c3e-4a91db85655f-openshift-service-ca\") pod \"maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.236576 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.236556 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-maas-api-tls\") pod \"maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.236616 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.236580 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-metrics-certs\") pod \"maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.266897 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.266865 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:12.390025 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.389927 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xfjr\" (UniqueName: \"kubernetes.io/projected/a44bddcd-a44c-4998-8c3e-4a91db85655f-kube-api-access-4xfjr\") pod \"maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.390310 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.390287 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44bddcd-a44c-4998-8c3e-4a91db85655f-openshift-service-ca\") pod \"maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.390569 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.390553 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-maas-api-tls\") pod \"maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.390681 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.390667 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-metrics-certs\") pod \"maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.391868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.391842 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44bddcd-a44c-4998-8c3e-4a91db85655f-openshift-service-ca\") pod \"maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.392182 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:26:12.392163 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-serving-cert: secret "maas-api-e2e-api-a-a9404c-serving-cert" not found Nov 03 00:26:12.392381 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:26:12.392351 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-maas-api-tls podName:a44bddcd-a44c-4998-8c3e-4a91db85655f nodeName:}" failed. No retries permitted until 2026-11-03 00:26:12.892329917 +0000 UTC m=+1219.843270053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-maas-api-tls") pod "maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" (UID: "a44bddcd-a44c-4998-8c3e-4a91db85655f") : secret "maas-api-e2e-api-a-a9404c-serving-cert" not found Nov 03 00:26:12.401339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.401316 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-metrics-certs\") pod \"maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.435623 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.435583 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:12.495597 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.495522 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xfjr\" (UniqueName: \"kubernetes.io/projected/a44bddcd-a44c-4998-8c3e-4a91db85655f-kube-api-access-4xfjr\") pod \"maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.770586 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.770443 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:12.902307 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.902277 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-maas-api-tls\") pod \"maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.904873 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.904797 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-maas-api-tls\") pod \"maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:12.969895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:12.969862 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:13.122072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:13.122031 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:13.346954 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:13.346930 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs"] Nov 03 00:26:13.348700 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:26:13.348673 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44bddcd_a44c_4998_8c3e_4a91db85655f.slice/crio-b7f83d416d92771d0f145557561d004ba7a868924b903821dc8e10b4adee28f0 WatchSource:0}: Error finding container b7f83d416d92771d0f145557561d004ba7a868924b903821dc8e10b4adee28f0: Status 404 returned error can't find the container with id b7f83d416d92771d0f145557561d004ba7a868924b903821dc8e10b4adee28f0 Nov 03 00:26:13.350160 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:13.350143 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:26:14.258140 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:14.258108 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" event={"ID":"a44bddcd-a44c-4998-8c3e-4a91db85655f","Type":"ContainerStarted","Data":"b7f83d416d92771d0f145557561d004ba7a868924b903821dc8e10b4adee28f0"} Nov 03 00:26:15.265240 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:15.265205 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" event={"ID":"a44bddcd-a44c-4998-8c3e-4a91db85655f","Type":"ContainerStarted","Data":"dfa7224a4fd791c3b47521ae3be5d0cb2a5cd50cb547779ae1dc27c1dd08a8be"} Nov 03 00:26:15.265751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:15.265392 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:15.382291 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:15.382239 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" podStartSLOduration=2.479559989 podStartE2EDuration="3.382223524s" totalImagesPullingTime="902.663535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:15.380766746 +0000 UTC m=+1222.331706905" watchObservedRunningTime="2026-11-03 00:26:15.382223524 +0000 UTC m=+1222.333163742" Nov 03 00:26:17.318417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:17.318379 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:21.282084 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:21.282058 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:26:22.171292 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:22.171253 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:26.536395 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.536353 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w"] Nov 03 00:26:26.540756 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.540739 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:26.596220 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.596187 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-openshift-service-ca\") pod \"maas-api-e2e-api-a-73821c-c8db94c7f-r799w\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:26.596384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.596240 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-maas-api-tls\") pod \"maas-api-e2e-api-a-73821c-c8db94c7f-r799w\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:26.596384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.596327 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-metrics-certs\") pod \"maas-api-e2e-api-a-73821c-c8db94c7f-r799w\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:26.596488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.596468 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmhw8\" (UniqueName: \"kubernetes.io/projected/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-kube-api-access-vmhw8\") pod \"maas-api-e2e-api-a-73821c-c8db94c7f-r799w\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:26.596656 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.596630 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w"] Nov 03 00:26:26.643112 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.643085 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-73821c-serving-cert\"" Nov 03 00:26:26.697330 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.697292 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmhw8\" (UniqueName: \"kubernetes.io/projected/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-kube-api-access-vmhw8\") pod \"maas-api-e2e-api-a-73821c-c8db94c7f-r799w\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:26.697508 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.697378 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-openshift-service-ca\") pod \"maas-api-e2e-api-a-73821c-c8db94c7f-r799w\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:26.697508 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.697418 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-maas-api-tls\") pod \"maas-api-e2e-api-a-73821c-c8db94c7f-r799w\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:26.697646 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.697528 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-metrics-certs\") pod \"maas-api-e2e-api-a-73821c-c8db94c7f-r799w\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:26.697867 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:26:26.697794 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-serving-cert: secret "maas-api-e2e-api-a-73821c-serving-cert" not found Nov 03 00:26:26.698008 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:26:26.697935 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-maas-api-tls podName:4f4b1f17-dada-4e8e-b63e-a43b76d73cd7 nodeName:}" failed. No retries permitted until 2026-11-03 00:26:27.197910135 +0000 UTC m=+1234.148850273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-maas-api-tls") pod "maas-api-e2e-api-a-73821c-c8db94c7f-r799w" (UID: "4f4b1f17-dada-4e8e-b63e-a43b76d73cd7") : secret "maas-api-e2e-api-a-73821c-serving-cert" not found Nov 03 00:26:26.698274 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.698249 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-openshift-service-ca\") pod \"maas-api-e2e-api-a-73821c-c8db94c7f-r799w\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:26.700672 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.700650 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-metrics-certs\") pod \"maas-api-e2e-api-a-73821c-c8db94c7f-r799w\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:26.754340 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.754313 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmhw8\" (UniqueName: \"kubernetes.io/projected/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-kube-api-access-vmhw8\") pod \"maas-api-e2e-api-a-73821c-c8db94c7f-r799w\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:26.789721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:26.789647 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:27.010843 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:27.010434 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:27.148622 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:27.148529 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:26:27.166314 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:26:27.166283 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:26:27.172215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:27.172189 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:27.204835 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:27.204793 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-maas-api-tls\") pod \"maas-api-e2e-api-a-73821c-c8db94c7f-r799w\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:27.207330 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:27.207303 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-maas-api-tls\") pod \"maas-api-e2e-api-a-73821c-c8db94c7f-r799w\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:27.454519 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:27.454441 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:27.873915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:27.873879 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w"] Nov 03 00:26:28.344336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:28.344296 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" event={"ID":"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7","Type":"ContainerStarted","Data":"22e8c1f7fa9b72b49363992060a3ca96757065c4ab5d16e2b8407cb35560ace3"} Nov 03 00:26:29.351286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:29.351244 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" event={"ID":"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7","Type":"ContainerStarted","Data":"f4b85b3bbef540ca6712cdf0a45a912605d9ad0db0c9acba7941d87d130ff530"} Nov 03 00:26:29.351728 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:29.351332 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:29.464419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:29.464355 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" podStartSLOduration=2.524280821 podStartE2EDuration="3.464338476s" totalImagesPullingTime="940.057655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:29.463588782 +0000 UTC m=+1236.414528940" watchObservedRunningTime="2026-11-03 00:26:29.464338476 +0000 UTC m=+1236.415278635" Nov 03 00:26:31.715053 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:31.715014 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:35.363396 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:35.363369 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:26:36.514089 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:36.514055 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:38.764260 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:38.764225 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:41.637118 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:41.637061 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:46.419367 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:46.419331 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:50.446961 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:50.446903 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:26:51.035352 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:51.035310 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:55.771168 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:55.771129 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:57.153351 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:26:57.153304 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:26:57.901858 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:57.901785 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs"] Nov 03 00:26:57.906013 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:57.905991 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:57.982315 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:57.982286 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs"] Nov 03 00:26:57.999156 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:57.999114 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-dshm\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:57.999156 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:57.999149 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-model-cache\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:57.999389 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:57.999264 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tls-certs\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:57.999469 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:57.999447 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-home\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:57.999579 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:57.999560 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5trfq\" (UniqueName: \"kubernetes.io/projected/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-kube-api-access-5trfq\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:57.999638 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:57.999597 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tmp-dir\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.009530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.009496 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cb65f3a5\"/\"e2e-lifecycle-model-cb65f3a5-kserve-self-signed-certs\"" Nov 03 00:26:58.009530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.009508 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cb65f3a5\"/\"default-dockercfg-hqnvp\"" Nov 03 00:26:58.009530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.009526 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cb65f3a5\"/\"kube-root-ca.crt\"" Nov 03 00:26:58.009915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.009901 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cb65f3a5\"/\"openshift-service-ca.crt\"" Nov 03 00:26:58.024411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.024385 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:58.105171 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.105128 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5trfq\" (UniqueName: \"kubernetes.io/projected/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-kube-api-access-5trfq\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.105857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.105804 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tmp-dir\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.106535 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.106515 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-dshm\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.107045 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.107024 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-model-cache\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.120912 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.107226 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tls-certs\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.121506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.121477 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-home\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.121743 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.106396 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tmp-dir\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.121875 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.111139 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-dshm\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.121979 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.113486 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tls-certs\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.122057 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.107334 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-model-cache\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.122411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.122393 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-home\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.168706 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.168609 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5trfq\" (UniqueName: \"kubernetes.io/projected/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-kube-api-access-5trfq\") pod \"e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.217993 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.217954 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:26:58.671242 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:58.671213 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs"] Nov 03 00:26:58.673018 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:26:58.672989 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ebaf2f2_ebc4_4baa_b326_d9c2491e1735.slice/crio-cbee76efd7eabc020755338b71a5e0ded68ab2e428b8b7225eceb218b4a8da1c WatchSource:0}: Error finding container cbee76efd7eabc020755338b71a5e0ded68ab2e428b8b7225eceb218b4a8da1c: Status 404 returned error can't find the container with id cbee76efd7eabc020755338b71a5e0ded68ab2e428b8b7225eceb218b4a8da1c Nov 03 00:26:59.571228 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:59.571194 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" event={"ID":"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735","Type":"ContainerStarted","Data":"db15702e925fa2159bbe4c345392203de94bf70219b52580ce9595f45f5c97bc"} Nov 03 00:26:59.571608 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:59.571236 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" event={"ID":"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735","Type":"ContainerStarted","Data":"cbee76efd7eabc020755338b71a5e0ded68ab2e428b8b7225eceb218b4a8da1c"} Nov 03 00:26:59.688924 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:26:59.688869 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" podStartSLOduration=2.688854455 podStartE2EDuration="2.688854455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:59.686507346 +0000 UTC m=+1266.637447503" watchObservedRunningTime="2026-11-03 00:26:59.688854455 +0000 UTC m=+1266.639794621" Nov 03 00:27:00.218047 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:00.218013 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:27:00.699241 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:00.699208 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:02.630848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:02.630739 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:03.560377 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:03.560342 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:04.221696 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:04.221663 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:04.515922 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:04.515809 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:05.450609 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:05.450576 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:05.817416 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:05.817386 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:08.218881 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:08.218840 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:27:08.225384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:08.225347 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:08.231588 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:08.231566 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:27:08.455583 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:08.455548 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:08.635892 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:08.635868 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:27:10.805537 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:10.805505 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:11.048043 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:11.048008 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:11.477596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:11.477562 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:11.659118 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:11.658939 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:13.897323 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:13.897283 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:14.624769 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:14.622433 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:14.826356 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:14.826322 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:16.092176 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:16.092140 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:18.237437 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:18.237403 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:18.959490 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:18.959459 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:19.379882 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:19.378577 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:19.604995 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:19.604954 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:19.784499 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:19.784464 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:20.262066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:20.261973 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:20.449644 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:20.449612 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:21.264381 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:21.264334 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:22.684349 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:22.684314 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:22.872006 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:22.871747 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:24.462740 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:24.462706 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:24.643880 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:24.643844 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:24.845635 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:24.845593 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:26.589389 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:26.589354 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:27.152671 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:27:27.152627 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:27:27.703156 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:27.703118 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:27.871547 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:27.871309 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:31.388118 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:31.388053 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:33.179409 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:33.179376 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:27:36.112556 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:36.112524 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:40.799510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:40.799476 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:45.483847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.483788 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs"] Nov 03 00:27:45.487799 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.487780 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.546091 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.546059 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs"] Nov 03 00:27:45.571048 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.571009 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-531ee4\"/\"kube-root-ca.crt\"" Nov 03 00:27:45.571048 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.571043 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-531ee4\"/\"openshift-service-ca.crt\"" Nov 03 00:27:45.591637 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.591601 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-531ee4\"/\"distinct-2-w2-531ee4-kserve-self-signed-certs\"" Nov 03 00:27:45.591637 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.591601 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-531ee4\"/\"default-dockercfg-br2w9\"" Nov 03 00:27:45.605741 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.605675 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:45.630189 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.630150 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tls-certs\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.630539 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.630514 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-model-cache\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.630790 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.630770 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-home\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.633066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.632742 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwfwn\" (UniqueName: \"kubernetes.io/projected/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-kube-api-access-kwfwn\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.643845 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.636456 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tmp-dir\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.668884 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.644320 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-dshm\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.771165 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.771062 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-dshm\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.771165 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.771126 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tls-certs\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.771417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.771176 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-model-cache\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.771417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.771209 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-home\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.771417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.771245 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwfwn\" (UniqueName: \"kubernetes.io/projected/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-kube-api-access-kwfwn\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.771417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.771337 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tmp-dir\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.771641 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.771618 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-model-cache\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.771700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.771662 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-home\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.771750 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.771708 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tmp-dir\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.774113 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.774066 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-dshm\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.774507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.774484 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tls-certs\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:45.827967 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.827934 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:45.830969 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:45.830943 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwfwn\" (UniqueName: \"kubernetes.io/projected/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-kube-api-access-kwfwn\") pod \"distinct-2-w2-531ee4-kserve-5ff474499-s4ghs\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:46.100600 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:46.100568 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:46.480913 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:46.480873 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs"] Nov 03 00:27:46.482382 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:27:46.482355 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfbe3d21_ff1e_4d7e_8e16_705478b6f95d.slice/crio-09a5ad586bb4d0f425d3ebc736d0cb2b2a08e4be93d31fc0e503b18e1547129e WatchSource:0}: Error finding container 09a5ad586bb4d0f425d3ebc736d0cb2b2a08e4be93d31fc0e503b18e1547129e: Status 404 returned error can't find the container with id 09a5ad586bb4d0f425d3ebc736d0cb2b2a08e4be93d31fc0e503b18e1547129e Nov 03 00:27:47.046414 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:47.046376 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" event={"ID":"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d","Type":"ContainerStarted","Data":"08761ea40ed934e9a1091ed36c7711bc77e932e044db9cf9fd88192d17e3a99b"} Nov 03 00:27:47.046850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:47.046423 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" event={"ID":"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d","Type":"ContainerStarted","Data":"09a5ad586bb4d0f425d3ebc736d0cb2b2a08e4be93d31fc0e503b18e1547129e"} Nov 03 00:27:47.101562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:47.101534 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:47.162342 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:47.162286 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" podStartSLOduration=2.162271413 podStartE2EDuration="2.162271413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:47.161043141 +0000 UTC m=+1314.111983300" watchObservedRunningTime="2026-11-03 00:27:47.162271413 +0000 UTC m=+1314.113211570" Nov 03 00:27:47.390199 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:47.390115 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:47.595311 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:47.595279 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:52.568433 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:52.568394 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:54.855253 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:54.855212 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8"] Nov 03 00:27:54.859387 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:54.859364 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:54.911393 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:54.911351 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e671ea-serving-cert\"" Nov 03 00:27:54.964434 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:54.964394 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8"] Nov 03 00:27:54.996930 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:54.996902 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-maas-api-tls\") pod \"maas-api-e2e-api-b-e671ea-7cdc456749-7qps8\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:54.997101 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:54.996957 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfdx6\" (UniqueName: \"kubernetes.io/projected/7094e012-a61d-4e4b-b8db-f42498a0e762-kube-api-access-dfdx6\") pod \"maas-api-e2e-api-b-e671ea-7cdc456749-7qps8\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:54.997169 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:54.997143 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-metrics-certs\") pod \"maas-api-e2e-api-b-e671ea-7cdc456749-7qps8\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:54.997304 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:54.997284 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7094e012-a61d-4e4b-b8db-f42498a0e762-openshift-service-ca\") pod \"maas-api-e2e-api-b-e671ea-7cdc456749-7qps8\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:55.077037 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:55.077006 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:55.099512 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:55.099471 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-metrics-certs\") pod \"maas-api-e2e-api-b-e671ea-7cdc456749-7qps8\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:55.100419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:55.100394 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7094e012-a61d-4e4b-b8db-f42498a0e762-openshift-service-ca\") pod \"maas-api-e2e-api-b-e671ea-7cdc456749-7qps8\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:55.101639 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:55.101617 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-maas-api-tls\") pod \"maas-api-e2e-api-b-e671ea-7cdc456749-7qps8\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:55.102002 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:55.101983 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfdx6\" (UniqueName: \"kubernetes.io/projected/7094e012-a61d-4e4b-b8db-f42498a0e762-kube-api-access-dfdx6\") pod \"maas-api-e2e-api-b-e671ea-7cdc456749-7qps8\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:55.102607 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:27:55.101879 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-serving-cert: secret "maas-api-e2e-api-b-e671ea-serving-cert" not found Nov 03 00:27:55.102851 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:27:55.102836 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-maas-api-tls podName:7094e012-a61d-4e4b-b8db-f42498a0e762 nodeName:}" failed. No retries permitted until 2026-11-03 00:27:55.602795675 +0000 UTC m=+1322.553735813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-maas-api-tls") pod "maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" (UID: "7094e012-a61d-4e4b-b8db-f42498a0e762") : secret "maas-api-e2e-api-b-e671ea-serving-cert" not found Nov 03 00:27:55.103423 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:55.103402 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7094e012-a61d-4e4b-b8db-f42498a0e762-openshift-service-ca\") pod \"maas-api-e2e-api-b-e671ea-7cdc456749-7qps8\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:55.125893 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:55.125763 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-metrics-certs\") pod \"maas-api-e2e-api-b-e671ea-7cdc456749-7qps8\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:55.177060 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:55.177011 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfdx6\" (UniqueName: \"kubernetes.io/projected/7094e012-a61d-4e4b-b8db-f42498a0e762-kube-api-access-dfdx6\") pod \"maas-api-e2e-api-b-e671ea-7cdc456749-7qps8\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:55.236574 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:55.236537 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:55.609502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:55.609467 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-maas-api-tls\") pod \"maas-api-e2e-api-b-e671ea-7cdc456749-7qps8\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:55.612099 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:55.612079 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-maas-api-tls\") pod \"maas-api-e2e-api-b-e671ea-7cdc456749-7qps8\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:55.692235 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:55.692204 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:55.773236 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:55.773202 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:56.101751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:56.101725 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:56.114262 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:56.114239 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:56.151384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:56.151363 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8"] Nov 03 00:27:56.152977 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:27:56.152953 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7094e012_a61d_4e4b_b8db_f42498a0e762.slice/crio-a2ec5d29b741eadb0bc51ea1874b0240df87e6f02c287c379c9b1a64736943ea WatchSource:0}: Error finding container a2ec5d29b741eadb0bc51ea1874b0240df87e6f02c287c379c9b1a64736943ea: Status 404 returned error can't find the container with id a2ec5d29b741eadb0bc51ea1874b0240df87e6f02c287c379c9b1a64736943ea Nov 03 00:27:57.111132 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:57.111100 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" event={"ID":"7094e012-a61d-4e4b-b8db-f42498a0e762","Type":"ContainerStarted","Data":"a2ec5d29b741eadb0bc51ea1874b0240df87e6f02c287c379c9b1a64736943ea"} Nov 03 00:27:57.117371 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:57.117349 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 00:27:57.170744 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:27:57.170709 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:27:58.119277 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:58.119225 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" event={"ID":"7094e012-a61d-4e4b-b8db-f42498a0e762","Type":"ContainerStarted","Data":"2a7f817de47be0ab4cc09e7cbcf6298e6760936872b79ffe23dfa1044ebbde58"} Nov 03 00:27:58.119899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:58.119315 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:27:58.235202 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:27:58.235148 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" podStartSLOduration=3.29813886 podStartE2EDuration="4.235134503s" totalImagesPullingTime="936.995643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:58.235080336 +0000 UTC m=+1325.186020497" watchObservedRunningTime="2026-11-03 00:27:58.235134503 +0000 UTC m=+1325.186074660" Nov 03 00:28:00.013830 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:00.013791 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:04.131438 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:04.131411 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:28:04.811169 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:04.811139 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:05.819025 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:05.818972 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:09.241483 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.241450 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d"] Nov 03 00:28:09.245475 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.245458 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:09.299777 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.299747 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-be578d-serving-cert\"" Nov 03 00:28:09.313860 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.313812 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-metrics-certs\") pod \"maas-api-e2e-api-a-be578d-654b6bc958-6mh4d\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:09.314017 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.313994 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3080ba7d-614b-4e1b-9d97-a8432e673152-openshift-service-ca\") pod \"maas-api-e2e-api-a-be578d-654b6bc958-6mh4d\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:09.314506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.314477 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-maas-api-tls\") pod \"maas-api-e2e-api-a-be578d-654b6bc958-6mh4d\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:09.314710 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.314679 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr47k\" (UniqueName: \"kubernetes.io/projected/3080ba7d-614b-4e1b-9d97-a8432e673152-kube-api-access-zr47k\") pod \"maas-api-e2e-api-a-be578d-654b6bc958-6mh4d\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:09.352267 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.352223 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d"] Nov 03 00:28:09.415750 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.415714 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-metrics-certs\") pod \"maas-api-e2e-api-a-be578d-654b6bc958-6mh4d\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:09.415933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.415807 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3080ba7d-614b-4e1b-9d97-a8432e673152-openshift-service-ca\") pod \"maas-api-e2e-api-a-be578d-654b6bc958-6mh4d\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:09.415933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.415859 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-maas-api-tls\") pod \"maas-api-e2e-api-a-be578d-654b6bc958-6mh4d\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:09.415933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.415884 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr47k\" (UniqueName: \"kubernetes.io/projected/3080ba7d-614b-4e1b-9d97-a8432e673152-kube-api-access-zr47k\") pod \"maas-api-e2e-api-a-be578d-654b6bc958-6mh4d\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:09.416081 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:28:09.415998 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-serving-cert: secret "maas-api-e2e-api-a-be578d-serving-cert" not found Nov 03 00:28:09.416081 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:28:09.416062 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-maas-api-tls podName:3080ba7d-614b-4e1b-9d97-a8432e673152 nodeName:}" failed. No retries permitted until 2026-11-03 00:28:09.916045158 +0000 UTC m=+1336.866985307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-maas-api-tls") pod "maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" (UID: "3080ba7d-614b-4e1b-9d97-a8432e673152") : secret "maas-api-e2e-api-a-be578d-serving-cert" not found Nov 03 00:28:09.416546 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.416524 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3080ba7d-614b-4e1b-9d97-a8432e673152-openshift-service-ca\") pod \"maas-api-e2e-api-a-be578d-654b6bc958-6mh4d\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:09.418366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.418346 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-metrics-certs\") pod \"maas-api-e2e-api-a-be578d-654b6bc958-6mh4d\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:09.458344 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.458311 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:09.474851 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.474788 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr47k\" (UniqueName: \"kubernetes.io/projected/3080ba7d-614b-4e1b-9d97-a8432e673152-kube-api-access-zr47k\") pod \"maas-api-e2e-api-a-be578d-654b6bc958-6mh4d\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:09.662649 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.662610 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:09.828593 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.828557 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:09.930887 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.930789 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-maas-api-tls\") pod \"maas-api-e2e-api-a-be578d-654b6bc958-6mh4d\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:09.933529 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:09.933497 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-maas-api-tls\") pod \"maas-api-e2e-api-a-be578d-654b6bc958-6mh4d\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:10.158498 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:10.158468 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:10.540458 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:10.540432 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d"] Nov 03 00:28:10.541975 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:28:10.541948 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3080ba7d_614b_4e1b_9d97_a8432e673152.slice/crio-c3b9af5727590c05630eb02f4b7d84794767dade770355fda71b503296e734c9 WatchSource:0}: Error finding container c3b9af5727590c05630eb02f4b7d84794767dade770355fda71b503296e734c9: Status 404 returned error can't find the container with id c3b9af5727590c05630eb02f4b7d84794767dade770355fda71b503296e734c9 Nov 03 00:28:11.196501 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:11.196463 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" event={"ID":"3080ba7d-614b-4e1b-9d97-a8432e673152","Type":"ContainerStarted","Data":"c3b9af5727590c05630eb02f4b7d84794767dade770355fda71b503296e734c9"} Nov 03 00:28:12.209340 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:12.209292 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" event={"ID":"3080ba7d-614b-4e1b-9d97-a8432e673152","Type":"ContainerStarted","Data":"8206ea1fb6bfe54b2bedd683d2fad66bd80139af8023303b850eb4583f5dc188"} Nov 03 00:28:12.209782 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:12.209392 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:12.333211 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:12.333127 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" podStartSLOduration=2.434963763 podStartE2EDuration="3.333100366s" totalImagesPullingTime="898.136603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:12.331297576 +0000 UTC m=+1339.282237734" watchObservedRunningTime="2026-11-03 00:28:12.333100366 +0000 UTC m=+1339.284040587" Nov 03 00:28:14.405144 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:14.405107 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:18.220113 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:18.220085 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:28:18.837232 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:18.837198 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn"] Nov 03 00:28:18.841712 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:18.841691 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:18.894765 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:18.894737 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-fd6ea3-serving-cert\"" Nov 03 00:28:18.913843 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:18.913803 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn"] Nov 03 00:28:18.949546 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:18.949508 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae5f286-eabe-4810-8c48-eca87ad5d06a-openshift-service-ca\") pod \"maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:18.949702 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:18.949581 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-metrics-certs\") pod \"maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:18.949771 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:18.949723 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh7wf\" (UniqueName: \"kubernetes.io/projected/1ae5f286-eabe-4810-8c48-eca87ad5d06a-kube-api-access-lh7wf\") pod \"maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:18.949853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:18.949794 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-maas-api-tls\") pod \"maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:19.051371 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:19.051342 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae5f286-eabe-4810-8c48-eca87ad5d06a-openshift-service-ca\") pod \"maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:19.051562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:19.051401 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-metrics-certs\") pod \"maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:19.051562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:19.051464 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh7wf\" (UniqueName: \"kubernetes.io/projected/1ae5f286-eabe-4810-8c48-eca87ad5d06a-kube-api-access-lh7wf\") pod \"maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:19.051562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:19.051513 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-maas-api-tls\") pod \"maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:19.051737 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:28:19.051694 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-serving-cert: secret "maas-api-e2e-api-b-fd6ea3-serving-cert" not found Nov 03 00:28:19.051797 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:28:19.051759 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-maas-api-tls podName:1ae5f286-eabe-4810-8c48-eca87ad5d06a nodeName:}" failed. No retries permitted until 2026-11-03 00:28:19.551739854 +0000 UTC m=+1346.502680005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-maas-api-tls") pod "maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" (UID: "1ae5f286-eabe-4810-8c48-eca87ad5d06a") : secret "maas-api-e2e-api-b-fd6ea3-serving-cert" not found Nov 03 00:28:19.052108 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:19.052074 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae5f286-eabe-4810-8c48-eca87ad5d06a-openshift-service-ca\") pod \"maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:19.054126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:19.054105 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-metrics-certs\") pod \"maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:19.080314 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:19.080287 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:19.110733 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:19.110665 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh7wf\" (UniqueName: \"kubernetes.io/projected/1ae5f286-eabe-4810-8c48-eca87ad5d06a-kube-api-access-lh7wf\") pod \"maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:19.303848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:19.296250 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:19.573303 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:19.573261 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-maas-api-tls\") pod \"maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:19.575870 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:19.575813 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-maas-api-tls\") pod \"maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:19.755714 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:19.755682 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:20.135497 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:20.135476 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn"] Nov 03 00:28:20.137258 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:28:20.137228 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae5f286_eabe_4810_8c48_eca87ad5d06a.slice/crio-a4189e8bde3dee0fa54e2065ee796436a589b1be59c373a7b6f1afc4f33ffca8 WatchSource:0}: Error finding container a4189e8bde3dee0fa54e2065ee796436a589b1be59c373a7b6f1afc4f33ffca8: Status 404 returned error can't find the container with id a4189e8bde3dee0fa54e2065ee796436a589b1be59c373a7b6f1afc4f33ffca8 Nov 03 00:28:20.258613 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:20.258584 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" event={"ID":"1ae5f286-eabe-4810-8c48-eca87ad5d06a","Type":"ContainerStarted","Data":"a4189e8bde3dee0fa54e2065ee796436a589b1be59c373a7b6f1afc4f33ffca8"} Nov 03 00:28:21.285946 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:21.279701 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" event={"ID":"1ae5f286-eabe-4810-8c48-eca87ad5d06a","Type":"ContainerStarted","Data":"a54df55b7bb5a710f721a60d51ebb8ccd22612ebd9905297425cccabf8f39a73"} Nov 03 00:28:21.285946 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:21.280884 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:21.360921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:21.360884 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:28:21.427239 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:21.427177 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" podStartSLOduration=2.63112803 podStartE2EDuration="3.427159091s" totalImagesPullingTime="796.031061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:21.423645436 +0000 UTC m=+1348.374585594" watchObservedRunningTime="2026-11-03 00:28:21.427159091 +0000 UTC m=+1348.378099249" Nov 03 00:28:24.097659 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:24.097616 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:27.152984 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:28:27.152949 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:28:28.297047 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:28.297019 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:28:28.914766 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:28.914732 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:33.511507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:33.511418 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:36.681068 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:36.681024 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:36.848333 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:36.848278 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:38.275203 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:38.275153 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:42.940099 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:42.940051 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:47.617196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:47.617151 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:52.363852 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:52.363800 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:57.153265 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:28:57.153226 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:28:57.222479 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:28:57.222434 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:00.489594 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:00.489563 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:01.962437 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:01.962400 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:02.220578 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:02.220490 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:04.072075 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:04.072040 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7306-80ece0"] Nov 03 00:29:04.077979 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:04.077960 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7306-80ece0" Nov 03 00:29:04.124192 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:04.124163 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prcnt\" (UniqueName: \"kubernetes.io/projected/96edf027-ad3b-4111-b2d0-fee13d263784-kube-api-access-prcnt\") pod \"test-curl-7306-80ece0\" (UID: \"96edf027-ad3b-4111-b2d0-fee13d263784\") " pod="opendatahub/test-curl-7306-80ece0" Nov 03 00:29:04.128813 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:04.128776 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7306-80ece0"] Nov 03 00:29:04.175291 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:04.175263 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fz96m\"" Nov 03 00:29:04.225484 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:04.225454 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prcnt\" (UniqueName: \"kubernetes.io/projected/96edf027-ad3b-4111-b2d0-fee13d263784-kube-api-access-prcnt\") pod \"test-curl-7306-80ece0\" (UID: \"96edf027-ad3b-4111-b2d0-fee13d263784\") " pod="opendatahub/test-curl-7306-80ece0" Nov 03 00:29:04.282335 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:04.282310 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prcnt\" (UniqueName: \"kubernetes.io/projected/96edf027-ad3b-4111-b2d0-fee13d263784-kube-api-access-prcnt\") pod \"test-curl-7306-80ece0\" (UID: \"96edf027-ad3b-4111-b2d0-fee13d263784\") " pod="opendatahub/test-curl-7306-80ece0" Nov 03 00:29:04.390394 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:04.390314 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7306-80ece0" Nov 03 00:29:04.762188 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:04.762158 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7306-80ece0"] Nov 03 00:29:04.764431 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:29:04.764400 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96edf027_ad3b_4111_b2d0_fee13d263784.slice/crio-1ddc2b0d708982a0743b05b48018f4d0e2b430920db2aa25de5e3c2fbb86ab11 WatchSource:0}: Error finding container 1ddc2b0d708982a0743b05b48018f4d0e2b430920db2aa25de5e3c2fbb86ab11: Status 404 returned error can't find the container with id 1ddc2b0d708982a0743b05b48018f4d0e2b430920db2aa25de5e3c2fbb86ab11 Nov 03 00:29:05.586095 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:05.586057 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7306-80ece0" event={"ID":"96edf027-ad3b-4111-b2d0-fee13d263784","Type":"ContainerStarted","Data":"1ddc2b0d708982a0743b05b48018f4d0e2b430920db2aa25de5e3c2fbb86ab11"} Nov 03 00:29:06.203190 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:06.203156 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:07.921753 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:07.921720 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:08.606354 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:08.606320 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7306-80ece0" event={"ID":"96edf027-ad3b-4111-b2d0-fee13d263784","Type":"ContainerStarted","Data":"8bc52a843a84cd806c022751f883b6239a269f41d9f03448a82f805cf870ef39"} Nov 03 00:29:08.716509 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:08.716450 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7306-80ece0" podStartSLOduration=1.772310104 podStartE2EDuration="4.716431315s" totalImagesPullingTime="2.944121211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:08.716217155 +0000 UTC m=+1395.667157312" watchObservedRunningTime="2026-11-03 00:29:08.716431315 +0000 UTC m=+1395.667371471" Nov 03 00:29:08.855734 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:08.855699 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:09.056597 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:09.056565 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:10.293847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:10.291360 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7306-80ece0"] Nov 03 00:29:10.322297 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:10.318099 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7306-80ece0"] Nov 03 00:29:10.619110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:10.619020 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7306-80ece0" podUID="96edf027-ad3b-4111-b2d0-fee13d263784" containerName="test-curl-7306-80ece0" containerID="cri-o://8bc52a843a84cd806c022751f883b6239a269f41d9f03448a82f805cf870ef39" gracePeriod=2 Nov 03 00:29:10.669587 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:29:10.669553 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7306-80ece0\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="96edf027-ad3b-4111-b2d0-fee13d263784" pod="opendatahub/test-curl-7306-80ece0" Nov 03 00:29:12.075438 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:12.075399 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:12.865216 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:12.865193 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7306-80ece0" Nov 03 00:29:12.953283 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:12.953218 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96edf027-ad3b-4111-b2d0-fee13d263784-kube-api-access-prcnt\" (UniqueName: \"kubernetes.io/projected/96edf027-ad3b-4111-b2d0-fee13d263784-kube-api-access-prcnt\") pod \"96edf027-ad3b-4111-b2d0-fee13d263784\" (UID: \"96edf027-ad3b-4111-b2d0-fee13d263784\") " Nov 03 00:29:12.955389 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:12.955364 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96edf027-ad3b-4111-b2d0-fee13d263784-kube-api-access-prcnt" pod "96edf027-ad3b-4111-b2d0-fee13d263784" (UID: "96edf027-ad3b-4111-b2d0-fee13d263784"). InnerVolumeSpecName "kube-api-access-prcnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:13.055301 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:13.055273 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prcnt\" (UniqueName: \"kubernetes.io/projected/96edf027-ad3b-4111-b2d0-fee13d263784-kube-api-access-prcnt\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:13.179135 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:13.179100 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96edf027-ad3b-4111-b2d0-fee13d263784" path="/var/lib/kubelet/pods/96edf027-ad3b-4111-b2d0-fee13d263784/volumes" Nov 03 00:29:13.607323 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:13.607286 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:13.643885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:13.643855 6400 generic.go:356] "Generic (PLEG): container finished" podID="96edf027-ad3b-4111-b2d0-fee13d263784" containerID="8bc52a843a84cd806c022751f883b6239a269f41d9f03448a82f805cf870ef39" exitCode=137 Nov 03 00:29:13.644066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:13.643893 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7306-80ece0" Nov 03 00:29:13.644066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:13.643949 6400 scope.go:122] "RemoveContainer" containerID="8bc52a843a84cd806c022751f883b6239a269f41d9f03448a82f805cf870ef39" Nov 03 00:29:13.654302 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:13.654284 6400 scope.go:122] "RemoveContainer" containerID="8bc52a843a84cd806c022751f883b6239a269f41d9f03448a82f805cf870ef39" Nov 03 00:29:13.654658 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:29:13.654638 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc52a843a84cd806c022751f883b6239a269f41d9f03448a82f805cf870ef39\": container with ID starting with 8bc52a843a84cd806c022751f883b6239a269f41d9f03448a82f805cf870ef39 not found: ID does not exist" containerID="8bc52a843a84cd806c022751f883b6239a269f41d9f03448a82f805cf870ef39" Nov 03 00:29:13.654750 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:13.654672 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc52a843a84cd806c022751f883b6239a269f41d9f03448a82f805cf870ef39"} err="failed to get container status \"8bc52a843a84cd806c022751f883b6239a269f41d9f03448a82f805cf870ef39\": rpc error: code = NotFound desc = could not find container \"8bc52a843a84cd806c022751f883b6239a269f41d9f03448a82f805cf870ef39\": container with ID starting with 8bc52a843a84cd806c022751f883b6239a269f41d9f03448a82f805cf870ef39 not found: ID does not exist" Nov 03 00:29:16.378492 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:16.378459 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:16.827570 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:16.827527 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:16.997762 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:16.997722 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:17.212380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:17.212248 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:17.389043 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:17.389012 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:17.565865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:17.565796 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:18.147271 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:18.147143 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:18.517998 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:18.517906 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:18.947176 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:18.947143 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:29:22.459099 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:22.459058 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:22.871219 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:22.871157 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:23.101339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:23.101303 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:23.281327 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:23.281225 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:23.479860 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:23.479790 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:23.669452 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:23.669418 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:23.849428 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:23.849381 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:26.222387 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:26.222353 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:26.450286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:26.449584 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:27.125871 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:27.125804 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:27.187997 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:29:27.187954 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:29:27.687926 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:27.687898 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:28.099298 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:28.099265 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:29.471895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:29.471861 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:29.796034 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:29.796001 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:30.005386 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:30.005353 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:32.385968 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:32.385928 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:32.591886 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:32.591693 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:32.778897 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:32.778753 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:35.380072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:35.380041 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:36.257511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:36.257480 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:36.477718 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:36.475754 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:36.782792 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:36.782761 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs"] Nov 03 00:29:36.783068 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:36.783043 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" podUID="a44bddcd-a44c-4998-8c3e-4a91db85655f" containerName="maas-api" containerID="cri-o://dfa7224a4fd791c3b47521ae3be5d0cb2a5cd50cb547779ae1dc27c1dd08a8be" gracePeriod=30 Nov 03 00:29:36.909015 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:36.908979 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:37.235857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.235808 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:29:37.250039 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.250005 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-maas-api-tls\") pod \"a44bddcd-a44c-4998-8c3e-4a91db85655f\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " Nov 03 00:29:37.250196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.250086 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44bddcd-a44c-4998-8c3e-4a91db85655f-kube-api-access-4xfjr\" (UniqueName: \"kubernetes.io/projected/a44bddcd-a44c-4998-8c3e-4a91db85655f-kube-api-access-4xfjr\") pod \"a44bddcd-a44c-4998-8c3e-4a91db85655f\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " Nov 03 00:29:37.250196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.250107 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44bddcd-a44c-4998-8c3e-4a91db85655f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44bddcd-a44c-4998-8c3e-4a91db85655f-openshift-service-ca\") pod \"a44bddcd-a44c-4998-8c3e-4a91db85655f\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " Nov 03 00:29:37.250345 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.250329 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-metrics-certs\") pod \"a44bddcd-a44c-4998-8c3e-4a91db85655f\" (UID: \"a44bddcd-a44c-4998-8c3e-4a91db85655f\") " Nov 03 00:29:37.250525 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.250494 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44bddcd-a44c-4998-8c3e-4a91db85655f-openshift-service-ca" pod "a44bddcd-a44c-4998-8c3e-4a91db85655f" (UID: "a44bddcd-a44c-4998-8c3e-4a91db85655f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:37.250807 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.250785 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44bddcd-a44c-4998-8c3e-4a91db85655f-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:37.252631 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.252605 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-maas-api-tls" pod "a44bddcd-a44c-4998-8c3e-4a91db85655f" (UID: "a44bddcd-a44c-4998-8c3e-4a91db85655f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:29:37.252854 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.252790 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44bddcd-a44c-4998-8c3e-4a91db85655f-kube-api-access-4xfjr" pod "a44bddcd-a44c-4998-8c3e-4a91db85655f" (UID: "a44bddcd-a44c-4998-8c3e-4a91db85655f"). InnerVolumeSpecName "kube-api-access-4xfjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:37.252974 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.252866 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-metrics-certs" pod "a44bddcd-a44c-4998-8c3e-4a91db85655f" (UID: "a44bddcd-a44c-4998-8c3e-4a91db85655f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:29:37.383170 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.383133 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:37.383411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.383396 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44bddcd-a44c-4998-8c3e-4a91db85655f-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:37.383510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.383496 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xfjr\" (UniqueName: \"kubernetes.io/projected/a44bddcd-a44c-4998-8c3e-4a91db85655f-kube-api-access-4xfjr\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:37.878573 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.878537 6400 generic.go:356] "Generic (PLEG): container finished" podID="a44bddcd-a44c-4998-8c3e-4a91db85655f" containerID="dfa7224a4fd791c3b47521ae3be5d0cb2a5cd50cb547779ae1dc27c1dd08a8be" exitCode=0 Nov 03 00:29:37.879123 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.879099 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" event={"ID":"a44bddcd-a44c-4998-8c3e-4a91db85655f","Type":"ContainerDied","Data":"dfa7224a4fd791c3b47521ae3be5d0cb2a5cd50cb547779ae1dc27c1dd08a8be"} Nov 03 00:29:37.879253 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.879240 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" event={"ID":"a44bddcd-a44c-4998-8c3e-4a91db85655f","Type":"ContainerDied","Data":"b7f83d416d92771d0f145557561d004ba7a868924b903821dc8e10b4adee28f0"} Nov 03 00:29:37.879347 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.879337 6400 scope.go:122] "RemoveContainer" containerID="dfa7224a4fd791c3b47521ae3be5d0cb2a5cd50cb547779ae1dc27c1dd08a8be" Nov 03 00:29:37.879579 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.879564 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs" Nov 03 00:29:37.960283 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.960250 6400 scope.go:122] "RemoveContainer" containerID="dfa7224a4fd791c3b47521ae3be5d0cb2a5cd50cb547779ae1dc27c1dd08a8be" Nov 03 00:29:37.961493 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:29:37.961461 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa7224a4fd791c3b47521ae3be5d0cb2a5cd50cb547779ae1dc27c1dd08a8be\": container with ID starting with dfa7224a4fd791c3b47521ae3be5d0cb2a5cd50cb547779ae1dc27c1dd08a8be not found: ID does not exist" containerID="dfa7224a4fd791c3b47521ae3be5d0cb2a5cd50cb547779ae1dc27c1dd08a8be" Nov 03 00:29:37.961687 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:37.961661 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa7224a4fd791c3b47521ae3be5d0cb2a5cd50cb547779ae1dc27c1dd08a8be"} err="failed to get container status \"dfa7224a4fd791c3b47521ae3be5d0cb2a5cd50cb547779ae1dc27c1dd08a8be\": rpc error: code = NotFound desc = could not find container \"dfa7224a4fd791c3b47521ae3be5d0cb2a5cd50cb547779ae1dc27c1dd08a8be\": container with ID starting with dfa7224a4fd791c3b47521ae3be5d0cb2a5cd50cb547779ae1dc27c1dd08a8be not found: ID does not exist" Nov 03 00:29:38.054072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:38.054038 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs"] Nov 03 00:29:38.103528 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:38.103483 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9404c-6b595c4444-zwjzs"] Nov 03 00:29:38.886418 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:38.886385 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:39.179180 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:39.179107 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44bddcd-a44c-4998-8c3e-4a91db85655f" path="/var/lib/kubelet/pods/a44bddcd-a44c-4998-8c3e-4a91db85655f/volumes" Nov 03 00:29:41.277477 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:41.277430 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:43.932219 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:43.932187 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:46.865470 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:46.865433 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:47.203379 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:47.203283 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:48.460028 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:48.459996 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:48.904810 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:48.904773 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:49.081293 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:49.081261 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:49.690678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:49.690636 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:53.528829 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:53.528783 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w"] Nov 03 00:29:53.529212 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:53.529018 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" podUID="4f4b1f17-dada-4e8e-b63e-a43b76d73cd7" containerName="maas-api" containerID="cri-o://f4b85b3bbef540ca6712cdf0a45a912605d9ad0db0c9acba7941d87d130ff530" gracePeriod=30 Nov 03 00:29:53.654855 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:53.654741 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:53.968196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:53.968118 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:29:54.017848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.016884 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-kube-api-access-vmhw8\" (UniqueName: \"kubernetes.io/projected/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-kube-api-access-vmhw8\") pod \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " Nov 03 00:29:54.017848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.017371 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-maas-api-tls\") pod \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " Nov 03 00:29:54.017848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.017497 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-metrics-certs\") pod \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " Nov 03 00:29:54.017848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.017646 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-openshift-service-ca\") pod \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\" (UID: \"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7\") " Nov 03 00:29:54.019531 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.018412 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-openshift-service-ca" pod "4f4b1f17-dada-4e8e-b63e-a43b76d73cd7" (UID: "4f4b1f17-dada-4e8e-b63e-a43b76d73cd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:54.041391 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.041167 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-metrics-certs" pod "4f4b1f17-dada-4e8e-b63e-a43b76d73cd7" (UID: "4f4b1f17-dada-4e8e-b63e-a43b76d73cd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:29:54.056904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.050022 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-kube-api-access-vmhw8" pod "4f4b1f17-dada-4e8e-b63e-a43b76d73cd7" (UID: "4f4b1f17-dada-4e8e-b63e-a43b76d73cd7"). InnerVolumeSpecName "kube-api-access-vmhw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:54.056904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.053366 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-maas-api-tls" pod "4f4b1f17-dada-4e8e-b63e-a43b76d73cd7" (UID: "4f4b1f17-dada-4e8e-b63e-a43b76d73cd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:29:54.069092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.067172 6400 generic.go:356] "Generic (PLEG): container finished" podID="4f4b1f17-dada-4e8e-b63e-a43b76d73cd7" containerID="f4b85b3bbef540ca6712cdf0a45a912605d9ad0db0c9acba7941d87d130ff530" exitCode=0 Nov 03 00:29:54.069092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.067367 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" event={"ID":"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7","Type":"ContainerDied","Data":"f4b85b3bbef540ca6712cdf0a45a912605d9ad0db0c9acba7941d87d130ff530"} Nov 03 00:29:54.069092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.067451 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" event={"ID":"4f4b1f17-dada-4e8e-b63e-a43b76d73cd7","Type":"ContainerDied","Data":"22e8c1f7fa9b72b49363992060a3ca96757065c4ab5d16e2b8407cb35560ace3"} Nov 03 00:29:54.069092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.067485 6400 scope.go:122] "RemoveContainer" containerID="f4b85b3bbef540ca6712cdf0a45a912605d9ad0db0c9acba7941d87d130ff530" Nov 03 00:29:54.069092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.067643 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w" Nov 03 00:29:54.086019 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.085682 6400 scope.go:122] "RemoveContainer" containerID="f4b85b3bbef540ca6712cdf0a45a912605d9ad0db0c9acba7941d87d130ff530" Nov 03 00:29:54.089843 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:29:54.086791 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b85b3bbef540ca6712cdf0a45a912605d9ad0db0c9acba7941d87d130ff530\": container with ID starting with f4b85b3bbef540ca6712cdf0a45a912605d9ad0db0c9acba7941d87d130ff530 not found: ID does not exist" containerID="f4b85b3bbef540ca6712cdf0a45a912605d9ad0db0c9acba7941d87d130ff530" Nov 03 00:29:54.089843 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.087893 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b85b3bbef540ca6712cdf0a45a912605d9ad0db0c9acba7941d87d130ff530"} err="failed to get container status \"f4b85b3bbef540ca6712cdf0a45a912605d9ad0db0c9acba7941d87d130ff530\": rpc error: code = NotFound desc = could not find container \"f4b85b3bbef540ca6712cdf0a45a912605d9ad0db0c9acba7941d87d130ff530\": container with ID starting with f4b85b3bbef540ca6712cdf0a45a912605d9ad0db0c9acba7941d87d130ff530 not found: ID does not exist" Nov 03 00:29:54.135885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.128466 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.135885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.128498 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmhw8\" (UniqueName: \"kubernetes.io/projected/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-kube-api-access-vmhw8\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.135885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.128513 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.135885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.128527 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.232771 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.232673 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w"] Nov 03 00:29:54.235553 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:54.235528 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73821c-c8db94c7f-r799w"] Nov 03 00:29:55.180726 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:55.180691 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f4b1f17-dada-4e8e-b63e-a43b76d73cd7" path="/var/lib/kubelet/pods/4f4b1f17-dada-4e8e-b63e-a43b76d73cd7/volumes" Nov 03 00:29:55.825146 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:55.825111 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:56.035336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:56.035306 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:56.803014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:56.802960 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:56.981815 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:56.979906 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:57.153336 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:29:57.153241 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:29:58.331164 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:58.331130 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:58.833287 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:58.833252 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:59.609353 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:29:59.609316 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:00.206297 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.206267 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m"] Nov 03 00:30:00.206902 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.206884 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4f4b1f17-dada-4e8e-b63e-a43b76d73cd7" containerName="maas-api" Nov 03 00:30:00.207023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.206904 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4b1f17-dada-4e8e-b63e-a43b76d73cd7" containerName="maas-api" Nov 03 00:30:00.207023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.206918 6400 cpu_manager.go:414] "containerMap: removing container" podUID="96edf027-ad3b-4111-b2d0-fee13d263784" containerName="test-curl-7306-80ece0" Nov 03 00:30:00.207023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.206925 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96edf027-ad3b-4111-b2d0-fee13d263784" containerName="test-curl-7306-80ece0" Nov 03 00:30:00.207023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.206986 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a44bddcd-a44c-4998-8c3e-4a91db85655f" containerName="maas-api" Nov 03 00:30:00.207023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.206994 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44bddcd-a44c-4998-8c3e-4a91db85655f" containerName="maas-api" Nov 03 00:30:00.207260 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.207147 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44bddcd-a44c-4998-8c3e-4a91db85655f" containerName="maas-api" Nov 03 00:30:00.207260 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.207165 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4b1f17-dada-4e8e-b63e-a43b76d73cd7" containerName="maas-api" Nov 03 00:30:00.207260 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.207174 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="96edf027-ad3b-4111-b2d0-fee13d263784" containerName="test-curl-7306-80ece0" Nov 03 00:30:00.211762 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.211742 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst"] Nov 03 00:30:00.211913 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.211873 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" Nov 03 00:30:00.215663 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.215641 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" Nov 03 00:30:00.330627 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.330589 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m"] Nov 03 00:30:00.330794 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.330674 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst"] Nov 03 00:30:00.357235 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.357189 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e624205d-ca8e-40eb-b1bd-135005deab0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894430-dqmst\" (UID: \"e624205d-ca8e-40eb-b1bd-135005deab0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" Nov 03 00:30:00.357424 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.357251 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bw5\" (UniqueName: \"kubernetes.io/projected/e624205d-ca8e-40eb-b1bd-135005deab0d-kube-api-access-c8bw5\") pod \"maas-api-key-cleanup-29894430-dqmst\" (UID: \"e624205d-ca8e-40eb-b1bd-135005deab0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" Nov 03 00:30:00.357531 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.357505 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ns5h\" (UniqueName: \"kubernetes.io/projected/19a02d3c-eaf0-4e95-991c-aa684c598248-kube-api-access-2ns5h\") pod \"maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m\" (UID: \"19a02d3c-eaf0-4e95-991c-aa684c598248\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" Nov 03 00:30:00.357684 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.357662 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19a02d3c-eaf0-4e95-991c-aa684c598248-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m\" (UID: \"19a02d3c-eaf0-4e95-991c-aa684c598248\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" Nov 03 00:30:00.371023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.370995 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:30:00.458869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.458781 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ns5h\" (UniqueName: \"kubernetes.io/projected/19a02d3c-eaf0-4e95-991c-aa684c598248-kube-api-access-2ns5h\") pod \"maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m\" (UID: \"19a02d3c-eaf0-4e95-991c-aa684c598248\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" Nov 03 00:30:00.459014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.458912 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19a02d3c-eaf0-4e95-991c-aa684c598248-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m\" (UID: \"19a02d3c-eaf0-4e95-991c-aa684c598248\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" Nov 03 00:30:00.459063 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.459022 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e624205d-ca8e-40eb-b1bd-135005deab0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894430-dqmst\" (UID: \"e624205d-ca8e-40eb-b1bd-135005deab0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" Nov 03 00:30:00.459063 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.459056 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bw5\" (UniqueName: \"kubernetes.io/projected/e624205d-ca8e-40eb-b1bd-135005deab0d-kube-api-access-c8bw5\") pod \"maas-api-key-cleanup-29894430-dqmst\" (UID: \"e624205d-ca8e-40eb-b1bd-135005deab0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" Nov 03 00:30:00.459509 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.459491 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19a02d3c-eaf0-4e95-991c-aa684c598248-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m\" (UID: \"19a02d3c-eaf0-4e95-991c-aa684c598248\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" Nov 03 00:30:00.459677 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.459660 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e624205d-ca8e-40eb-b1bd-135005deab0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894430-dqmst\" (UID: \"e624205d-ca8e-40eb-b1bd-135005deab0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" Nov 03 00:30:00.516255 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.516224 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bw5\" (UniqueName: \"kubernetes.io/projected/e624205d-ca8e-40eb-b1bd-135005deab0d-kube-api-access-c8bw5\") pod \"maas-api-key-cleanup-29894430-dqmst\" (UID: \"e624205d-ca8e-40eb-b1bd-135005deab0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" Nov 03 00:30:00.516401 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.516277 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ns5h\" (UniqueName: \"kubernetes.io/projected/19a02d3c-eaf0-4e95-991c-aa684c598248-kube-api-access-2ns5h\") pod \"maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m\" (UID: \"19a02d3c-eaf0-4e95-991c-aa684c598248\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" Nov 03 00:30:00.529205 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.529177 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" Nov 03 00:30:00.538469 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.538439 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" Nov 03 00:30:00.927163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.927139 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m"] Nov 03 00:30:00.928839 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:30:00.928800 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a02d3c_eaf0_4e95_991c_aa684c598248.slice/crio-58ea62ad750df1c7a56daf28acdcd026bd3e6586ef95c4c940f3f3de8583d72a WatchSource:0}: Error finding container 58ea62ad750df1c7a56daf28acdcd026bd3e6586ef95c4c940f3f3de8583d72a: Status 404 returned error can't find the container with id 58ea62ad750df1c7a56daf28acdcd026bd3e6586ef95c4c940f3f3de8583d72a Nov 03 00:30:00.979701 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:00.979671 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst"] Nov 03 00:30:00.981701 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:30:00.981677 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode624205d_ca8e_40eb_b1bd_135005deab0d.slice/crio-1c25e686ca0078a84d82db625ed36b8a940a94db3a9d4333812c3d9cc39173b9 WatchSource:0}: Error finding container 1c25e686ca0078a84d82db625ed36b8a940a94db3a9d4333812c3d9cc39173b9: Status 404 returned error can't find the container with id 1c25e686ca0078a84d82db625ed36b8a940a94db3a9d4333812c3d9cc39173b9 Nov 03 00:30:01.049629 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:01.049593 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:01.155308 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:01.145710 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" event={"ID":"19a02d3c-eaf0-4e95-991c-aa684c598248","Type":"ContainerStarted","Data":"58ea62ad750df1c7a56daf28acdcd026bd3e6586ef95c4c940f3f3de8583d72a"} Nov 03 00:30:01.155308 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:01.147179 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" event={"ID":"e624205d-ca8e-40eb-b1bd-135005deab0d","Type":"ContainerStarted","Data":"1c25e686ca0078a84d82db625ed36b8a940a94db3a9d4333812c3d9cc39173b9"} Nov 03 00:30:01.708472 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:01.708437 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:02.154422 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:02.154385 6400 generic.go:356] "Generic (PLEG): container finished" podID="19a02d3c-eaf0-4e95-991c-aa684c598248" containerID="5cbd9c82dd62c4584948e2fc9d15f540e2f43ef51c8eabbf310dae0f4f771804" exitCode=60 Nov 03 00:30:02.154852 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:02.154433 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" event={"ID":"19a02d3c-eaf0-4e95-991c-aa684c598248","Type":"ContainerDied","Data":"5cbd9c82dd62c4584948e2fc9d15f540e2f43ef51c8eabbf310dae0f4f771804"} Nov 03 00:30:02.154852 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:02.154768 6400 scope.go:122] "RemoveContainer" containerID="5cbd9c82dd62c4584948e2fc9d15f540e2f43ef51c8eabbf310dae0f4f771804" Nov 03 00:30:02.156303 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:02.156275 6400 generic.go:356] "Generic (PLEG): container finished" podID="e624205d-ca8e-40eb-b1bd-135005deab0d" containerID="26751feeef86323607463f2ac954dae6dc65d714173921947a02f9ec243f02cd" exitCode=60 Nov 03 00:30:02.156414 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:02.156305 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" event={"ID":"e624205d-ca8e-40eb-b1bd-135005deab0d","Type":"ContainerDied","Data":"26751feeef86323607463f2ac954dae6dc65d714173921947a02f9ec243f02cd"} Nov 03 00:30:02.156613 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:02.156596 6400 scope.go:122] "RemoveContainer" containerID="26751feeef86323607463f2ac954dae6dc65d714173921947a02f9ec243f02cd" Nov 03 00:30:03.164365 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:03.164334 6400 generic.go:356] "Generic (PLEG): container finished" podID="e624205d-ca8e-40eb-b1bd-135005deab0d" containerID="108bc02f915b1c0a9d5e4a5d245b3edc487170959472755f1cc1900b653ba51d" exitCode=60 Nov 03 00:30:03.164789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:03.164423 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" event={"ID":"e624205d-ca8e-40eb-b1bd-135005deab0d","Type":"ContainerDied","Data":"108bc02f915b1c0a9d5e4a5d245b3edc487170959472755f1cc1900b653ba51d"} Nov 03 00:30:03.164789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:03.164504 6400 scope.go:122] "RemoveContainer" containerID="26751feeef86323607463f2ac954dae6dc65d714173921947a02f9ec243f02cd" Nov 03 00:30:03.164789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:03.164779 6400 scope.go:122] "RemoveContainer" containerID="108bc02f915b1c0a9d5e4a5d245b3edc487170959472755f1cc1900b653ba51d" Nov 03 00:30:03.165148 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:03.165128 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894430-dqmst_odh-ai-gateway-infra(e624205d-ca8e-40eb-b1bd-135005deab0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" podUID="e624205d-ca8e-40eb-b1bd-135005deab0d" Nov 03 00:30:03.166562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:03.166542 6400 generic.go:356] "Generic (PLEG): container finished" podID="19a02d3c-eaf0-4e95-991c-aa684c598248" containerID="870f21dac587a58147bd8d6e2252d1d0595acd6c5b9bedb43c6a28be6022d1e8" exitCode=60 Nov 03 00:30:03.166652 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:03.166580 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" event={"ID":"19a02d3c-eaf0-4e95-991c-aa684c598248","Type":"ContainerDied","Data":"870f21dac587a58147bd8d6e2252d1d0595acd6c5b9bedb43c6a28be6022d1e8"} Nov 03 00:30:03.166870 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:03.166855 6400 scope.go:122] "RemoveContainer" containerID="870f21dac587a58147bd8d6e2252d1d0595acd6c5b9bedb43c6a28be6022d1e8" Nov 03 00:30:03.167111 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:03.167096 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m_odh-ai-gateway-infra(19a02d3c-eaf0-4e95-991c-aa684c598248)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" podUID="19a02d3c-eaf0-4e95-991c-aa684c598248" Nov 03 00:30:03.176629 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:03.176607 6400 scope.go:122] "RemoveContainer" containerID="5cbd9c82dd62c4584948e2fc9d15f540e2f43ef51c8eabbf310dae0f4f771804" Nov 03 00:30:04.174280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:04.174254 6400 scope.go:122] "RemoveContainer" containerID="870f21dac587a58147bd8d6e2252d1d0595acd6c5b9bedb43c6a28be6022d1e8" Nov 03 00:30:04.174724 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:04.174576 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m_odh-ai-gateway-infra(19a02d3c-eaf0-4e95-991c-aa684c598248)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" podUID="19a02d3c-eaf0-4e95-991c-aa684c598248" Nov 03 00:30:04.175882 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:04.175861 6400 scope.go:122] "RemoveContainer" containerID="108bc02f915b1c0a9d5e4a5d245b3edc487170959472755f1cc1900b653ba51d" Nov 03 00:30:04.176185 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:04.176152 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894430-dqmst_odh-ai-gateway-infra(e624205d-ca8e-40eb-b1bd-135005deab0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" podUID="e624205d-ca8e-40eb-b1bd-135005deab0d" Nov 03 00:30:05.547138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:05.547106 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn"] Nov 03 00:30:05.547556 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:05.547329 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" podUID="1ae5f286-eabe-4810-8c48-eca87ad5d06a" containerName="maas-api" containerID="cri-o://a54df55b7bb5a710f721a60d51ebb8ccd22612ebd9905297425cccabf8f39a73" gracePeriod=30 Nov 03 00:30:05.673030 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:05.672993 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:05.933586 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:05.933538 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:30:06.068842 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.067904 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae5f286-eabe-4810-8c48-eca87ad5d06a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae5f286-eabe-4810-8c48-eca87ad5d06a-openshift-service-ca\") pod \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " Nov 03 00:30:06.068842 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.068060 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae5f286-eabe-4810-8c48-eca87ad5d06a-kube-api-access-lh7wf\" (UniqueName: \"kubernetes.io/projected/1ae5f286-eabe-4810-8c48-eca87ad5d06a-kube-api-access-lh7wf\") pod \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " Nov 03 00:30:06.068842 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.068113 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-metrics-certs\") pod \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " Nov 03 00:30:06.068842 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.068249 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-maas-api-tls\") pod \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\" (UID: \"1ae5f286-eabe-4810-8c48-eca87ad5d06a\") " Nov 03 00:30:06.073882 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.071253 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae5f286-eabe-4810-8c48-eca87ad5d06a-openshift-service-ca" pod "1ae5f286-eabe-4810-8c48-eca87ad5d06a" (UID: "1ae5f286-eabe-4810-8c48-eca87ad5d06a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:06.104632 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.104582 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-metrics-certs" pod "1ae5f286-eabe-4810-8c48-eca87ad5d06a" (UID: "1ae5f286-eabe-4810-8c48-eca87ad5d06a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:30:06.147923 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.141574 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-maas-api-tls" pod "1ae5f286-eabe-4810-8c48-eca87ad5d06a" (UID: "1ae5f286-eabe-4810-8c48-eca87ad5d06a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:30:06.147923 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.147466 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae5f286-eabe-4810-8c48-eca87ad5d06a-kube-api-access-lh7wf" pod "1ae5f286-eabe-4810-8c48-eca87ad5d06a" (UID: "1ae5f286-eabe-4810-8c48-eca87ad5d06a"). InnerVolumeSpecName "kube-api-access-lh7wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:06.170243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.170211 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh7wf\" (UniqueName: \"kubernetes.io/projected/1ae5f286-eabe-4810-8c48-eca87ad5d06a-kube-api-access-lh7wf\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:06.170243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.170241 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:06.170243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.170252 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae5f286-eabe-4810-8c48-eca87ad5d06a-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:06.170613 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.170261 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae5f286-eabe-4810-8c48-eca87ad5d06a-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:06.191662 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.191631 6400 generic.go:356] "Generic (PLEG): container finished" podID="1ae5f286-eabe-4810-8c48-eca87ad5d06a" containerID="a54df55b7bb5a710f721a60d51ebb8ccd22612ebd9905297425cccabf8f39a73" exitCode=0 Nov 03 00:30:06.191813 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.191687 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" event={"ID":"1ae5f286-eabe-4810-8c48-eca87ad5d06a","Type":"ContainerDied","Data":"a54df55b7bb5a710f721a60d51ebb8ccd22612ebd9905297425cccabf8f39a73"} Nov 03 00:30:06.191813 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.191725 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" event={"ID":"1ae5f286-eabe-4810-8c48-eca87ad5d06a","Type":"ContainerDied","Data":"a4189e8bde3dee0fa54e2065ee796436a589b1be59c373a7b6f1afc4f33ffca8"} Nov 03 00:30:06.191813 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.191743 6400 scope.go:122] "RemoveContainer" containerID="a54df55b7bb5a710f721a60d51ebb8ccd22612ebd9905297425cccabf8f39a73" Nov 03 00:30:06.192036 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.191896 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn" Nov 03 00:30:06.205003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.204940 6400 scope.go:122] "RemoveContainer" containerID="a54df55b7bb5a710f721a60d51ebb8ccd22612ebd9905297425cccabf8f39a73" Nov 03 00:30:06.205288 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:06.205269 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54df55b7bb5a710f721a60d51ebb8ccd22612ebd9905297425cccabf8f39a73\": container with ID starting with a54df55b7bb5a710f721a60d51ebb8ccd22612ebd9905297425cccabf8f39a73 not found: ID does not exist" containerID="a54df55b7bb5a710f721a60d51ebb8ccd22612ebd9905297425cccabf8f39a73" Nov 03 00:30:06.205343 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.205299 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54df55b7bb5a710f721a60d51ebb8ccd22612ebd9905297425cccabf8f39a73"} err="failed to get container status \"a54df55b7bb5a710f721a60d51ebb8ccd22612ebd9905297425cccabf8f39a73\": rpc error: code = NotFound desc = could not find container \"a54df55b7bb5a710f721a60d51ebb8ccd22612ebd9905297425cccabf8f39a73\": container with ID starting with a54df55b7bb5a710f721a60d51ebb8ccd22612ebd9905297425cccabf8f39a73 not found: ID does not exist" Nov 03 00:30:06.258615 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.258585 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:06.378844 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.368733 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn"] Nov 03 00:30:06.378844 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.375479 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fd6ea3-54dcbcff5-4v6hn"] Nov 03 00:30:06.433511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.433426 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:06.581758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:06.581724 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:07.180316 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.180280 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae5f286-eabe-4810-8c48-eca87ad5d06a" path="/var/lib/kubelet/pods/1ae5f286-eabe-4810-8c48-eca87ad5d06a/volumes" Nov 03 00:30:07.273411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.273354 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld"] Nov 03 00:30:07.274022 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.274003 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1ae5f286-eabe-4810-8c48-eca87ad5d06a" containerName="maas-api" Nov 03 00:30:07.274161 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.274023 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae5f286-eabe-4810-8c48-eca87ad5d06a" containerName="maas-api" Nov 03 00:30:07.274235 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.274221 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae5f286-eabe-4810-8c48-eca87ad5d06a" containerName="maas-api" Nov 03 00:30:07.278646 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.278626 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:07.288694 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.288669 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-metrics-certs\") pod \"maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:07.288853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.288713 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-maas-api-tls\") pod \"maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:07.288853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.288842 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87be3afe-a55b-48d2-a890-2788ea461965-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:07.288979 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.288959 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n82v2\" (UniqueName: \"kubernetes.io/projected/87be3afe-a55b-48d2-a890-2788ea461965-kube-api-access-n82v2\") pod \"maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:07.331061 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.331038 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-38f5a3-serving-cert\"" Nov 03 00:30:07.383029 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.382998 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld"] Nov 03 00:30:07.390095 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.390069 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n82v2\" (UniqueName: \"kubernetes.io/projected/87be3afe-a55b-48d2-a890-2788ea461965-kube-api-access-n82v2\") pod \"maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:07.390219 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.390190 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-metrics-certs\") pod \"maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:07.390279 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.390241 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-maas-api-tls\") pod \"maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:07.390345 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.390329 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87be3afe-a55b-48d2-a890-2788ea461965-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:07.390403 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:07.390391 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-serving-cert: secret "maas-api-e2e-ipp-a-38f5a3-serving-cert" not found Nov 03 00:30:07.390457 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:07.390449 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-maas-api-tls podName:87be3afe-a55b-48d2-a890-2788ea461965 nodeName:}" failed. No retries permitted until 2026-11-03 00:30:07.890433409 +0000 UTC m=+1454.841373562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-maas-api-tls") pod "maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" (UID: "87be3afe-a55b-48d2-a890-2788ea461965") : secret "maas-api-e2e-ipp-a-38f5a3-serving-cert" not found Nov 03 00:30:07.391030 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.391004 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87be3afe-a55b-48d2-a890-2788ea461965-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:07.393175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.393153 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-metrics-certs\") pod \"maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:07.447319 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.447252 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n82v2\" (UniqueName: \"kubernetes.io/projected/87be3afe-a55b-48d2-a890-2788ea461965-kube-api-access-n82v2\") pod \"maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:07.502445 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.502416 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:07.722901 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.720386 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:07.888978 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.888944 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:07.909448 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.909425 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-maas-api-tls\") pod \"maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:07.911939 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:07.911914 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-maas-api-tls\") pod \"maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:08.105858 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:08.105812 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:30:08.192193 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:08.192161 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:08.571605 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:08.571578 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld"] Nov 03 00:30:08.574074 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:30:08.573983 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87be3afe_a55b_48d2_a890_2788ea461965.slice/crio-6ec5350cf64ed3d46d5f86ce4a231c2558526e5b2512370d68c1fffb3fa00fe9 WatchSource:0}: Error finding container 6ec5350cf64ed3d46d5f86ce4a231c2558526e5b2512370d68c1fffb3fa00fe9: Status 404 returned error can't find the container with id 6ec5350cf64ed3d46d5f86ce4a231c2558526e5b2512370d68c1fffb3fa00fe9 Nov 03 00:30:09.213116 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:09.213080 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" event={"ID":"87be3afe-a55b-48d2-a890-2788ea461965","Type":"ContainerStarted","Data":"6ec5350cf64ed3d46d5f86ce4a231c2558526e5b2512370d68c1fffb3fa00fe9"} Nov 03 00:30:10.221704 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:10.221672 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" event={"ID":"87be3afe-a55b-48d2-a890-2788ea461965","Type":"ContainerStarted","Data":"1af4a5d9a71ea35ebf47583531c2662ff4cf29f7db6b6003f081bd606192c1c0"} Nov 03 00:30:10.222196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:10.221779 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:10.338035 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:10.337983 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" podStartSLOduration=2.396496156 podStartE2EDuration="3.337968555s" totalImagesPullingTime="941.472399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:10.336170361 +0000 UTC m=+1457.287110518" watchObservedRunningTime="2026-11-03 00:30:10.337968555 +0000 UTC m=+1457.288908742" Nov 03 00:30:11.203759 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:11.203727 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5"] Nov 03 00:30:11.203959 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:11.203931 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" podUID="23271791-3a3a-4695-aa8d-33362c295435" containerName="istio-proxy" containerID="cri-o://9260f57af9fd3e2c1308eb0d20f6c8abdb87fdb4e4354e1e1d5c2aaa7d81fd98" gracePeriod=30 Nov 03 00:30:12.168737 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.168696 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:12.640137 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.640105 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr"] Nov 03 00:30:12.644126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.644110 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:12.710274 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.710239 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-maas-api-tls\") pod \"maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:12.710448 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.710318 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l656q\" (UniqueName: \"kubernetes.io/projected/08e5e225-44ad-49a7-8b30-94ed44e3e344-kube-api-access-l656q\") pod \"maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:12.710521 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.710491 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-metrics-certs\") pod \"maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:12.710711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.710691 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e5e225-44ad-49a7-8b30-94ed44e3e344-openshift-service-ca\") pod \"maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:12.747080 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.747048 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c521ba-serving-cert\"" Nov 03 00:30:12.749708 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.749687 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr"] Nov 03 00:30:12.811840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.811799 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l656q\" (UniqueName: \"kubernetes.io/projected/08e5e225-44ad-49a7-8b30-94ed44e3e344-kube-api-access-l656q\") pod \"maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:12.811988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.811914 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-metrics-certs\") pod \"maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:12.812041 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.812027 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e5e225-44ad-49a7-8b30-94ed44e3e344-openshift-service-ca\") pod \"maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:12.812094 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.812045 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-maas-api-tls\") pod \"maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:12.812155 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:12.812138 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-serving-cert: secret "maas-api-e2e-api-b-c521ba-serving-cert" not found Nov 03 00:30:12.812208 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:12.812193 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-maas-api-tls podName:08e5e225-44ad-49a7-8b30-94ed44e3e344 nodeName:}" failed. No retries permitted until 2026-11-03 00:30:13.312174654 +0000 UTC m=+1460.263114807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-maas-api-tls") pod "maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" (UID: "08e5e225-44ad-49a7-8b30-94ed44e3e344") : secret "maas-api-e2e-api-b-c521ba-serving-cert" not found Nov 03 00:30:12.812665 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.812642 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e5e225-44ad-49a7-8b30-94ed44e3e344-openshift-service-ca\") pod \"maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:12.814336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.814321 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-metrics-certs\") pod \"maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:12.845371 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.845344 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:12.870482 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:12.870462 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l656q\" (UniqueName: \"kubernetes.io/projected/08e5e225-44ad-49a7-8b30-94ed44e3e344-kube-api-access-l656q\") pod \"maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:13.018569 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:13.018479 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:13.219042 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:13.218995 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:13.327310 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:13.320173 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-maas-api-tls\") pod \"maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:13.327310 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:13.324226 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-maas-api-tls\") pod \"maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:13.557581 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:13.557549 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:13.937000 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:13.936970 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr"] Nov 03 00:30:13.938412 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:30:13.938386 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e5e225_44ad_49a7_8b30_94ed44e3e344.slice/crio-d3b5f4a50a410c4f8f80918a1cd417685ee531259a112edd2c8723a14ad9030b WatchSource:0}: Error finding container d3b5f4a50a410c4f8f80918a1cd417685ee531259a112edd2c8723a14ad9030b: Status 404 returned error can't find the container with id d3b5f4a50a410c4f8f80918a1cd417685ee531259a112edd2c8723a14ad9030b Nov 03 00:30:14.245953 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:14.245883 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" event={"ID":"08e5e225-44ad-49a7-8b30-94ed44e3e344","Type":"ContainerStarted","Data":"d3b5f4a50a410c4f8f80918a1cd417685ee531259a112edd2c8723a14ad9030b"} Nov 03 00:30:15.252882 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:15.252849 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" event={"ID":"08e5e225-44ad-49a7-8b30-94ed44e3e344","Type":"ContainerStarted","Data":"b0a56b970371910ed276eda9b32931fec8c6ede07304faa1021c005f25131cde"} Nov 03 00:30:15.253252 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:15.252982 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:15.371527 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:15.371476 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" podStartSLOduration=2.575132455 podStartE2EDuration="3.371462569s" totalImagesPullingTime="796.330114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:15.369281891 +0000 UTC m=+1462.320222050" watchObservedRunningTime="2026-11-03 00:30:15.371462569 +0000 UTC m=+1462.322402726" Nov 03 00:30:16.233059 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.233040 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:30:16.498497 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.498474 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:30:16.664132 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664107 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-data\") pod \"23271791-3a3a-4695-aa8d-33362c295435\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " Nov 03 00:30:16.664278 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664139 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-socket\") pod \"23271791-3a3a-4695-aa8d-33362c295435\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " Nov 03 00:30:16.664278 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664205 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-kube-api-access-ltt2m\" (UniqueName: \"kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-kube-api-access-ltt2m\") pod \"23271791-3a3a-4695-aa8d-33362c295435\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " Nov 03 00:30:16.664278 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664252 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23271791-3a3a-4695-aa8d-33362c295435-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23271791-3a3a-4695-aa8d-33362c295435-istio-podinfo\") pod \"23271791-3a3a-4695-aa8d-33362c295435\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " Nov 03 00:30:16.664419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664285 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-credential-socket\") pod \"23271791-3a3a-4695-aa8d-33362c295435\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " Nov 03 00:30:16.664419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664310 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23271791-3a3a-4695-aa8d-33362c295435-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23271791-3a3a-4695-aa8d-33362c295435-istiod-ca-cert\") pod \"23271791-3a3a-4695-aa8d-33362c295435\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " Nov 03 00:30:16.664419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664382 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-istio-token\" (UniqueName: \"kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-istio-token\") pod \"23271791-3a3a-4695-aa8d-33362c295435\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " Nov 03 00:30:16.664419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664386 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-data" pod "23271791-3a3a-4695-aa8d-33362c295435" (UID: "23271791-3a3a-4695-aa8d-33362c295435"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:16.664596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664449 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-socket" pod "23271791-3a3a-4695-aa8d-33362c295435" (UID: "23271791-3a3a-4695-aa8d-33362c295435"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:16.664596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664519 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-certs\") pod \"23271791-3a3a-4695-aa8d-33362c295435\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " Nov 03 00:30:16.664596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664539 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-envoy\") pod \"23271791-3a3a-4695-aa8d-33362c295435\" (UID: \"23271791-3a3a-4695-aa8d-33362c295435\") " Nov 03 00:30:16.664744 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664619 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-credential-socket" pod "23271791-3a3a-4695-aa8d-33362c295435" (UID: "23271791-3a3a-4695-aa8d-33362c295435"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:16.664855 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664815 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-certs" pod "23271791-3a3a-4695-aa8d-33362c295435" (UID: "23271791-3a3a-4695-aa8d-33362c295435"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:16.664956 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.664864 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23271791-3a3a-4695-aa8d-33362c295435-istiod-ca-cert" pod "23271791-3a3a-4695-aa8d-33362c295435" (UID: "23271791-3a3a-4695-aa8d-33362c295435"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:16.665380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.665276 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.665380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.665301 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.665380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.665317 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.665380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.665330 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.665380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.665346 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23271791-3a3a-4695-aa8d-33362c295435-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.666840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.666804 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-envoy" pod "23271791-3a3a-4695-aa8d-33362c295435" (UID: "23271791-3a3a-4695-aa8d-33362c295435"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:16.666922 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.666835 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23271791-3a3a-4695-aa8d-33362c295435-istio-podinfo" pod "23271791-3a3a-4695-aa8d-33362c295435" (UID: "23271791-3a3a-4695-aa8d-33362c295435"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:30:16.666922 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.666874 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-istio-token" pod "23271791-3a3a-4695-aa8d-33362c295435" (UID: "23271791-3a3a-4695-aa8d-33362c295435"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:16.666922 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.666889 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-kube-api-access-ltt2m" pod "23271791-3a3a-4695-aa8d-33362c295435" (UID: "23271791-3a3a-4695-aa8d-33362c295435"). InnerVolumeSpecName "kube-api-access-ltt2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:16.766902 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.766812 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23271791-3a3a-4695-aa8d-33362c295435-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.766902 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.766862 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltt2m\" (UniqueName: \"kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-kube-api-access-ltt2m\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.766902 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.766873 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23271791-3a3a-4695-aa8d-33362c295435-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:16.766902 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:16.766881 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23271791-3a3a-4695-aa8d-33362c295435-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:17.172632 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:17.172607 6400 scope.go:122] "RemoveContainer" containerID="870f21dac587a58147bd8d6e2252d1d0595acd6c5b9bedb43c6a28be6022d1e8" Nov 03 00:30:17.264038 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:17.264011 6400 generic.go:356] "Generic (PLEG): container finished" podID="23271791-3a3a-4695-aa8d-33362c295435" containerID="9260f57af9fd3e2c1308eb0d20f6c8abdb87fdb4e4354e1e1d5c2aaa7d81fd98" exitCode=0 Nov 03 00:30:17.264140 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:17.264055 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" Nov 03 00:30:17.264140 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:17.264094 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" event={"ID":"23271791-3a3a-4695-aa8d-33362c295435","Type":"ContainerDied","Data":"9260f57af9fd3e2c1308eb0d20f6c8abdb87fdb4e4354e1e1d5c2aaa7d81fd98"} Nov 03 00:30:17.264218 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:17.264144 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5" event={"ID":"23271791-3a3a-4695-aa8d-33362c295435","Type":"ContainerDied","Data":"f40b8202c4308dc938e8d2d5352ebcbbe17e3440ba92296c3332b144a6829d22"} Nov 03 00:30:17.264218 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:17.264165 6400 scope.go:122] "RemoveContainer" containerID="9260f57af9fd3e2c1308eb0d20f6c8abdb87fdb4e4354e1e1d5c2aaa7d81fd98" Nov 03 00:30:17.273638 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:17.273620 6400 scope.go:122] "RemoveContainer" containerID="9260f57af9fd3e2c1308eb0d20f6c8abdb87fdb4e4354e1e1d5c2aaa7d81fd98" Nov 03 00:30:17.273921 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:17.273901 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9260f57af9fd3e2c1308eb0d20f6c8abdb87fdb4e4354e1e1d5c2aaa7d81fd98\": container with ID starting with 9260f57af9fd3e2c1308eb0d20f6c8abdb87fdb4e4354e1e1d5c2aaa7d81fd98 not found: ID does not exist" containerID="9260f57af9fd3e2c1308eb0d20f6c8abdb87fdb4e4354e1e1d5c2aaa7d81fd98" Nov 03 00:30:17.273974 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:17.273929 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9260f57af9fd3e2c1308eb0d20f6c8abdb87fdb4e4354e1e1d5c2aaa7d81fd98"} err="failed to get container status \"9260f57af9fd3e2c1308eb0d20f6c8abdb87fdb4e4354e1e1d5c2aaa7d81fd98\": rpc error: code = NotFound desc = could not find container \"9260f57af9fd3e2c1308eb0d20f6c8abdb87fdb4e4354e1e1d5c2aaa7d81fd98\": container with ID starting with 9260f57af9fd3e2c1308eb0d20f6c8abdb87fdb4e4354e1e1d5c2aaa7d81fd98 not found: ID does not exist" Nov 03 00:30:17.431704 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:17.430109 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5"] Nov 03 00:30:17.478118 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:17.478094 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-jdwq5"] Nov 03 00:30:17.817318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:17.817285 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:18.172763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:18.172688 6400 scope.go:122] "RemoveContainer" containerID="108bc02f915b1c0a9d5e4a5d245b3edc487170959472755f1cc1900b653ba51d" Nov 03 00:30:18.271711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:18.271679 6400 generic.go:356] "Generic (PLEG): container finished" podID="19a02d3c-eaf0-4e95-991c-aa684c598248" containerID="762581f91331640223fab426fe7bb1f5a26b250da168dd6d8dfaddd0a8e967bc" exitCode=60 Nov 03 00:30:18.271894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:18.271767 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" event={"ID":"19a02d3c-eaf0-4e95-991c-aa684c598248","Type":"ContainerDied","Data":"762581f91331640223fab426fe7bb1f5a26b250da168dd6d8dfaddd0a8e967bc"} Nov 03 00:30:18.271894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:18.271870 6400 scope.go:122] "RemoveContainer" containerID="870f21dac587a58147bd8d6e2252d1d0595acd6c5b9bedb43c6a28be6022d1e8" Nov 03 00:30:18.272235 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:18.272219 6400 scope.go:122] "RemoveContainer" containerID="762581f91331640223fab426fe7bb1f5a26b250da168dd6d8dfaddd0a8e967bc" Nov 03 00:30:18.272520 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:18.272498 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m_odh-ai-gateway-infra(19a02d3c-eaf0-4e95-991c-aa684c598248)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" podUID="19a02d3c-eaf0-4e95-991c-aa684c598248" Nov 03 00:30:18.314948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:18.314926 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:18.496144 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:18.496074 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:19.179815 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.179785 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23271791-3a3a-4695-aa8d-33362c295435" path="/var/lib/kubelet/pods/23271791-3a3a-4695-aa8d-33362c295435/volumes" Nov 03 00:30:19.281696 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.281662 6400 generic.go:356] "Generic (PLEG): container finished" podID="e624205d-ca8e-40eb-b1bd-135005deab0d" containerID="52fd8314e34fe73499baa571f6a688c3677a3f1e1d4f07b4ad2dc239d80a0131" exitCode=60 Nov 03 00:30:19.281865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.281726 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" event={"ID":"e624205d-ca8e-40eb-b1bd-135005deab0d","Type":"ContainerDied","Data":"52fd8314e34fe73499baa571f6a688c3677a3f1e1d4f07b4ad2dc239d80a0131"} Nov 03 00:30:19.281865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.281779 6400 scope.go:122] "RemoveContainer" containerID="108bc02f915b1c0a9d5e4a5d245b3edc487170959472755f1cc1900b653ba51d" Nov 03 00:30:19.282163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.282143 6400 scope.go:122] "RemoveContainer" containerID="52fd8314e34fe73499baa571f6a688c3677a3f1e1d4f07b4ad2dc239d80a0131" Nov 03 00:30:19.282432 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:19.282416 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894430-dqmst_odh-ai-gateway-infra(e624205d-ca8e-40eb-b1bd-135005deab0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" podUID="e624205d-ca8e-40eb-b1bd-135005deab0d" Nov 03 00:30:19.403924 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.403856 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m"] Nov 03 00:30:19.584613 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.584588 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" Nov 03 00:30:19.710422 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.710351 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a02d3c-eaf0-4e95-991c-aa684c598248-kube-api-access-2ns5h\" (UniqueName: \"kubernetes.io/projected/19a02d3c-eaf0-4e95-991c-aa684c598248-kube-api-access-2ns5h\") pod \"19a02d3c-eaf0-4e95-991c-aa684c598248\" (UID: \"19a02d3c-eaf0-4e95-991c-aa684c598248\") " Nov 03 00:30:19.710586 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.710470 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a02d3c-eaf0-4e95-991c-aa684c598248-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19a02d3c-eaf0-4e95-991c-aa684c598248-maas-api-service-ca\") pod \"19a02d3c-eaf0-4e95-991c-aa684c598248\" (UID: \"19a02d3c-eaf0-4e95-991c-aa684c598248\") " Nov 03 00:30:19.710918 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.710880 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a02d3c-eaf0-4e95-991c-aa684c598248-maas-api-service-ca" pod "19a02d3c-eaf0-4e95-991c-aa684c598248" (UID: "19a02d3c-eaf0-4e95-991c-aa684c598248"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:19.711078 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.711060 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19a02d3c-eaf0-4e95-991c-aa684c598248-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:19.713150 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.713130 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a02d3c-eaf0-4e95-991c-aa684c598248-kube-api-access-2ns5h" pod "19a02d3c-eaf0-4e95-991c-aa684c598248" (UID: "19a02d3c-eaf0-4e95-991c-aa684c598248"). InnerVolumeSpecName "kube-api-access-2ns5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:19.812092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.812071 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ns5h\" (UniqueName: \"kubernetes.io/projected/19a02d3c-eaf0-4e95-991c-aa684c598248-kube-api-access-2ns5h\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:19.865080 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:19.865055 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:20.294138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.294108 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" Nov 03 00:30:20.294650 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.294107 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m" event={"ID":"19a02d3c-eaf0-4e95-991c-aa684c598248","Type":"ContainerDied","Data":"58ea62ad750df1c7a56daf28acdcd026bd3e6586ef95c4c940f3f3de8583d72a"} Nov 03 00:30:20.294807 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.294791 6400 scope.go:122] "RemoveContainer" containerID="762581f91331640223fab426fe7bb1f5a26b250da168dd6d8dfaddd0a8e967bc" Nov 03 00:30:20.360641 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.360500 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:20.502846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.457920 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst"] Nov 03 00:30:20.517874 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.516253 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m"] Nov 03 00:30:20.523853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.520380 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e671ea-29894430-tkq8m"] Nov 03 00:30:20.737870 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.737815 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" Nov 03 00:30:20.846254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.846219 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e624205d-ca8e-40eb-b1bd-135005deab0d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e624205d-ca8e-40eb-b1bd-135005deab0d-maas-api-service-ca\") pod \"e624205d-ca8e-40eb-b1bd-135005deab0d\" (UID: \"e624205d-ca8e-40eb-b1bd-135005deab0d\") " Nov 03 00:30:20.846388 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.846270 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e624205d-ca8e-40eb-b1bd-135005deab0d-kube-api-access-c8bw5\" (UniqueName: \"kubernetes.io/projected/e624205d-ca8e-40eb-b1bd-135005deab0d-kube-api-access-c8bw5\") pod \"e624205d-ca8e-40eb-b1bd-135005deab0d\" (UID: \"e624205d-ca8e-40eb-b1bd-135005deab0d\") " Nov 03 00:30:20.846618 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.846591 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e624205d-ca8e-40eb-b1bd-135005deab0d-maas-api-service-ca" pod "e624205d-ca8e-40eb-b1bd-135005deab0d" (UID: "e624205d-ca8e-40eb-b1bd-135005deab0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:20.846750 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.846737 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e624205d-ca8e-40eb-b1bd-135005deab0d-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:20.848410 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.848392 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e624205d-ca8e-40eb-b1bd-135005deab0d-kube-api-access-c8bw5" pod "e624205d-ca8e-40eb-b1bd-135005deab0d" (UID: "e624205d-ca8e-40eb-b1bd-135005deab0d"). InnerVolumeSpecName "kube-api-access-c8bw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:20.947427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:20.947361 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8bw5\" (UniqueName: \"kubernetes.io/projected/e624205d-ca8e-40eb-b1bd-135005deab0d-kube-api-access-c8bw5\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:21.178592 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:21.178557 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a02d3c-eaf0-4e95-991c-aa684c598248" path="/var/lib/kubelet/pods/19a02d3c-eaf0-4e95-991c-aa684c598248/volumes" Nov 03 00:30:21.263856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:21.263776 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:30:21.289365 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:21.289342 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:21.300443 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:21.300425 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" Nov 03 00:30:21.300856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:21.300469 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst" event={"ID":"e624205d-ca8e-40eb-b1bd-135005deab0d","Type":"ContainerDied","Data":"1c25e686ca0078a84d82db625ed36b8a940a94db3a9d4333812c3d9cc39173b9"} Nov 03 00:30:21.300856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:21.300525 6400 scope.go:122] "RemoveContainer" containerID="52fd8314e34fe73499baa571f6a688c3677a3f1e1d4f07b4ad2dc239d80a0131" Nov 03 00:30:21.489253 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:21.489223 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst"] Nov 03 00:30:21.502019 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:21.501992 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-dqmst"] Nov 03 00:30:21.940843 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:21.934848 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:22.361762 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:22.361728 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:22.998459 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:22.998432 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:23.179027 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:23.179001 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e624205d-ca8e-40eb-b1bd-135005deab0d" path="/var/lib/kubelet/pods/e624205d-ca8e-40eb-b1bd-135005deab0d/volumes" Nov 03 00:30:25.826559 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:25.826523 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:26.007110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:26.007076 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:26.185758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:26.185684 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:27.151979 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:27.151936 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:30:27.997498 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:27.997465 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:29.088136 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:29.088109 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:29.290241 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:29.290215 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:33.206146 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:33.206113 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:37.042531 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:37.042504 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:30:37.953688 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:37.953652 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:41.920913 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:41.920881 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8"] Nov 03 00:30:41.921340 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:41.921071 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" podUID="7094e012-a61d-4e4b-b8db-f42498a0e762" containerName="maas-api" containerID="cri-o://2a7f817de47be0ab4cc09e7cbcf6298e6760936872b79ffe23dfa1044ebbde58" gracePeriod=30 Nov 03 00:30:42.040744 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.040709 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:42.293297 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.293269 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:30:42.390020 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.389983 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-metrics-certs\") pod \"7094e012-a61d-4e4b-b8db-f42498a0e762\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " Nov 03 00:30:42.390227 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.390208 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7094e012-a61d-4e4b-b8db-f42498a0e762-kube-api-access-dfdx6\" (UniqueName: \"kubernetes.io/projected/7094e012-a61d-4e4b-b8db-f42498a0e762-kube-api-access-dfdx6\") pod \"7094e012-a61d-4e4b-b8db-f42498a0e762\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " Nov 03 00:30:42.390298 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.390245 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-maas-api-tls\") pod \"7094e012-a61d-4e4b-b8db-f42498a0e762\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " Nov 03 00:30:42.390298 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.390276 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7094e012-a61d-4e4b-b8db-f42498a0e762-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7094e012-a61d-4e4b-b8db-f42498a0e762-openshift-service-ca\") pod \"7094e012-a61d-4e4b-b8db-f42498a0e762\" (UID: \"7094e012-a61d-4e4b-b8db-f42498a0e762\") " Nov 03 00:30:42.390689 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.390655 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7094e012-a61d-4e4b-b8db-f42498a0e762-openshift-service-ca" pod "7094e012-a61d-4e4b-b8db-f42498a0e762" (UID: "7094e012-a61d-4e4b-b8db-f42498a0e762"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:42.390909 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.390809 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7094e012-a61d-4e4b-b8db-f42498a0e762-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:42.392607 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.392523 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7094e012-a61d-4e4b-b8db-f42498a0e762-kube-api-access-dfdx6" pod "7094e012-a61d-4e4b-b8db-f42498a0e762" (UID: "7094e012-a61d-4e4b-b8db-f42498a0e762"). InnerVolumeSpecName "kube-api-access-dfdx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:42.393064 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.393037 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-maas-api-tls" pod "7094e012-a61d-4e4b-b8db-f42498a0e762" (UID: "7094e012-a61d-4e4b-b8db-f42498a0e762"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:30:42.393293 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.393265 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-metrics-certs" pod "7094e012-a61d-4e4b-b8db-f42498a0e762" (UID: "7094e012-a61d-4e4b-b8db-f42498a0e762"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:30:42.422084 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.422049 6400 generic.go:356] "Generic (PLEG): container finished" podID="7094e012-a61d-4e4b-b8db-f42498a0e762" containerID="2a7f817de47be0ab4cc09e7cbcf6298e6760936872b79ffe23dfa1044ebbde58" exitCode=0 Nov 03 00:30:42.422262 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.422094 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" Nov 03 00:30:42.422262 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.422131 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" event={"ID":"7094e012-a61d-4e4b-b8db-f42498a0e762","Type":"ContainerDied","Data":"2a7f817de47be0ab4cc09e7cbcf6298e6760936872b79ffe23dfa1044ebbde58"} Nov 03 00:30:42.422262 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.422185 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8" event={"ID":"7094e012-a61d-4e4b-b8db-f42498a0e762","Type":"ContainerDied","Data":"a2ec5d29b741eadb0bc51ea1874b0240df87e6f02c287c379c9b1a64736943ea"} Nov 03 00:30:42.422262 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.422212 6400 scope.go:122] "RemoveContainer" containerID="2a7f817de47be0ab4cc09e7cbcf6298e6760936872b79ffe23dfa1044ebbde58" Nov 03 00:30:42.432663 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.432647 6400 scope.go:122] "RemoveContainer" containerID="2a7f817de47be0ab4cc09e7cbcf6298e6760936872b79ffe23dfa1044ebbde58" Nov 03 00:30:42.432991 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:42.432968 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7f817de47be0ab4cc09e7cbcf6298e6760936872b79ffe23dfa1044ebbde58\": container with ID starting with 2a7f817de47be0ab4cc09e7cbcf6298e6760936872b79ffe23dfa1044ebbde58 not found: ID does not exist" containerID="2a7f817de47be0ab4cc09e7cbcf6298e6760936872b79ffe23dfa1044ebbde58" Nov 03 00:30:42.433096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.433001 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7f817de47be0ab4cc09e7cbcf6298e6760936872b79ffe23dfa1044ebbde58"} err="failed to get container status \"2a7f817de47be0ab4cc09e7cbcf6298e6760936872b79ffe23dfa1044ebbde58\": rpc error: code = NotFound desc = could not find container \"2a7f817de47be0ab4cc09e7cbcf6298e6760936872b79ffe23dfa1044ebbde58\": container with ID starting with 2a7f817de47be0ab4cc09e7cbcf6298e6760936872b79ffe23dfa1044ebbde58 not found: ID does not exist" Nov 03 00:30:42.492575 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.492542 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfdx6\" (UniqueName: \"kubernetes.io/projected/7094e012-a61d-4e4b-b8db-f42498a0e762-kube-api-access-dfdx6\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:42.492575 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.492577 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:42.492790 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.492588 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7094e012-a61d-4e4b-b8db-f42498a0e762-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:42.616847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.614696 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8"] Nov 03 00:30:42.618994 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:42.618965 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e671ea-7cdc456749-7qps8"] Nov 03 00:30:43.186231 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:43.186199 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7094e012-a61d-4e4b-b8db-f42498a0e762" path="/var/lib/kubelet/pods/7094e012-a61d-4e4b-b8db-f42498a0e762/volumes" Nov 03 00:30:43.278865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:43.278811 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:44.899580 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:44.899549 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:45.107506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:45.107478 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:45.295371 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:45.295334 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:49.491205 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:49.491159 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:50.233720 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:50.233692 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:51.477065 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:51.477034 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:51.663471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:51.655401 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:53.047309 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:53.047275 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:53.546922 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:53.546881 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:53.724893 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:53.724500 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:53.893407 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:53.893374 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:55.421917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:55.421883 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:55.910776 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:55.910742 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:56.111502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:30:56.111459 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:57.151779 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:30:57.151744 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:31:00.597275 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:00.597246 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:04.418326 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:04.418290 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:05.128413 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:05.128377 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:05.360409 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:05.360363 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:05.954906 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:05.954455 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:06.123312 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:06.123282 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:06.299882 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:06.299850 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:06.926812 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:06.926775 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:09.356645 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:09.356612 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:09.519097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:09.519058 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:10.077228 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:10.077060 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:10.436245 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:10.436157 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:10.666556 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:10.666518 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:10.834755 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:10.834713 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:11.209876 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:11.209776 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:11.743092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:11.743055 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:12.285052 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:12.285019 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp"] Nov 03 00:31:12.285546 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:12.285486 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" podUID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" containerName="istio-proxy" containerID="cri-o://b1a09acd73a9ecdb949bb69282d1fd01f2ad2a2ca37dc4a075dd2886a7b5a70d" gracePeriod=30 Nov 03 00:31:12.335176 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:12.335150 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc"] Nov 03 00:31:12.335386 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:12.335367 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" podUID="57107b1c-15e3-448e-9dee-de481f4d4bca" containerName="istio-proxy" containerID="cri-o://64ab45fab60f0318861a1f5ac72780be7598778e51d6299c5738b76a51d86b3a" gracePeriod=30 Nov 03 00:31:12.337320 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:12.337301 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp"] Nov 03 00:31:12.337499 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:12.337473 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" podUID="d85c2951-14fc-4025-99b5-d78d7794cd0c" containerName="istio-proxy" containerID="cri-o://69e6c156d02e4c0fabfb7ed84ec0cbca50e9efd11558419e874998a4288f0fc8" gracePeriod=30 Nov 03 00:31:13.746083 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:13.746028 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:13.945545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:13.929369 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:15.126284 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:15.126244 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:15.353312 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:15.353285 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:16.328231 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:16.328201 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:16.494094 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:16.494061 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:17.270130 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.270097 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:31:17.270294 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.270153 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" podUID="57107b1c-15e3-448e-9dee-de481f4d4bca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:31:17.621014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.620981 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:31:17.661127 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.661107 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:31:17.664061 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.664042 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:31:17.749738 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.749706 6400 generic.go:356] "Generic (PLEG): container finished" podID="57107b1c-15e3-448e-9dee-de481f4d4bca" containerID="64ab45fab60f0318861a1f5ac72780be7598778e51d6299c5738b76a51d86b3a" exitCode=0 Nov 03 00:31:17.749904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.749753 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" Nov 03 00:31:17.749904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.749783 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" event={"ID":"57107b1c-15e3-448e-9dee-de481f4d4bca","Type":"ContainerDied","Data":"64ab45fab60f0318861a1f5ac72780be7598778e51d6299c5738b76a51d86b3a"} Nov 03 00:31:17.749904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.749861 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc" event={"ID":"57107b1c-15e3-448e-9dee-de481f4d4bca","Type":"ContainerDied","Data":"fab073acaca6304b5efcaf4afe5cbc4f5594d12c24008dd9256d6528ec67a0a4"} Nov 03 00:31:17.749904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.749899 6400 scope.go:122] "RemoveContainer" containerID="64ab45fab60f0318861a1f5ac72780be7598778e51d6299c5738b76a51d86b3a" Nov 03 00:31:17.751070 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.751052 6400 generic.go:356] "Generic (PLEG): container finished" podID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" containerID="b1a09acd73a9ecdb949bb69282d1fd01f2ad2a2ca37dc4a075dd2886a7b5a70d" exitCode=0 Nov 03 00:31:17.751166 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.751088 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" Nov 03 00:31:17.751166 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.751131 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" event={"ID":"46f7ad4c-9382-4007-96ea-06ed4b82a7ca","Type":"ContainerDied","Data":"b1a09acd73a9ecdb949bb69282d1fd01f2ad2a2ca37dc4a075dd2886a7b5a70d"} Nov 03 00:31:17.751445 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.751171 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp" event={"ID":"46f7ad4c-9382-4007-96ea-06ed4b82a7ca","Type":"ContainerDied","Data":"faf0198f874eae91837b567a8f7d54e06d2743179e2cda57d0b1aee032ae69ed"} Nov 03 00:31:17.752649 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.752623 6400 generic.go:356] "Generic (PLEG): container finished" podID="d85c2951-14fc-4025-99b5-d78d7794cd0c" containerID="69e6c156d02e4c0fabfb7ed84ec0cbca50e9efd11558419e874998a4288f0fc8" exitCode=0 Nov 03 00:31:17.752768 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.752683 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" event={"ID":"d85c2951-14fc-4025-99b5-d78d7794cd0c","Type":"ContainerDied","Data":"69e6c156d02e4c0fabfb7ed84ec0cbca50e9efd11558419e874998a4288f0fc8"} Nov 03 00:31:17.752768 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.752716 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" event={"ID":"d85c2951-14fc-4025-99b5-d78d7794cd0c","Type":"ContainerDied","Data":"3edf0e6b388e95106f7490e99d0585e89f5f4f31def170ff67188de97c9d8aea"} Nov 03 00:31:17.752768 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.752718 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp" Nov 03 00:31:17.758917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.758898 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-socket\") pod \"57107b1c-15e3-448e-9dee-de481f4d4bca\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " Nov 03 00:31:17.759017 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.758937 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istiod-ca-cert\") pod \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " Nov 03 00:31:17.759017 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.758961 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-certs\") pod \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " Nov 03 00:31:17.759017 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.758997 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-socket\") pod \"d85c2951-14fc-4025-99b5-d78d7794cd0c\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " Nov 03 00:31:17.759185 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759024 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85c2951-14fc-4025-99b5-d78d7794cd0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85c2951-14fc-4025-99b5-d78d7794cd0c-istiod-ca-cert\") pod \"d85c2951-14fc-4025-99b5-d78d7794cd0c\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " Nov 03 00:31:17.759185 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759041 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-socket\") pod \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " Nov 03 00:31:17.759185 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759066 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57107b1c-15e3-448e-9dee-de481f4d4bca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57107b1c-15e3-448e-9dee-de481f4d4bca-istiod-ca-cert\") pod \"57107b1c-15e3-448e-9dee-de481f4d4bca\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " Nov 03 00:31:17.759185 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759130 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-credential-socket\") pod \"57107b1c-15e3-448e-9dee-de481f4d4bca\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " Nov 03 00:31:17.759185 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759156 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-data\") pod \"57107b1c-15e3-448e-9dee-de481f4d4bca\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " Nov 03 00:31:17.759185 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759183 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-credential-socket\") pod \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " Nov 03 00:31:17.759397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759202 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-socket" pod "57107b1c-15e3-448e-9dee-de481f4d4bca" (UID: "57107b1c-15e3-448e-9dee-de481f4d4bca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.759397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759222 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-certs\") pod \"57107b1c-15e3-448e-9dee-de481f4d4bca\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " Nov 03 00:31:17.759397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759292 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-certs\") pod \"d85c2951-14fc-4025-99b5-d78d7794cd0c\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " Nov 03 00:31:17.759397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759296 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-socket" pod "d85c2951-14fc-4025-99b5-d78d7794cd0c" (UID: "d85c2951-14fc-4025-99b5-d78d7794cd0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.759397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759318 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-data\") pod \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " Nov 03 00:31:17.759397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759338 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istiod-ca-cert" pod "46f7ad4c-9382-4007-96ea-06ed4b82a7ca" (UID: "46f7ad4c-9382-4007-96ea-06ed4b82a7ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:17.759397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759356 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-token\" (UniqueName: \"kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-token\") pod \"d85c2951-14fc-4025-99b5-d78d7794cd0c\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " Nov 03 00:31:17.759397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759384 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-data\") pod \"d85c2951-14fc-4025-99b5-d78d7794cd0c\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " Nov 03 00:31:17.759626 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759452 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-kube-api-access-2bnfg\" (UniqueName: \"kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-kube-api-access-2bnfg\") pod \"57107b1c-15e3-448e-9dee-de481f4d4bca\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " Nov 03 00:31:17.759626 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759485 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-envoy\") pod \"d85c2951-14fc-4025-99b5-d78d7794cd0c\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " Nov 03 00:31:17.759626 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759518 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-kube-api-access-4nlj2\" (UniqueName: \"kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-kube-api-access-4nlj2\") pod \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " Nov 03 00:31:17.759626 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759443 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85c2951-14fc-4025-99b5-d78d7794cd0c-istiod-ca-cert" pod "d85c2951-14fc-4025-99b5-d78d7794cd0c" (UID: "d85c2951-14fc-4025-99b5-d78d7794cd0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:17.759626 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759559 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-podinfo\") pod \"d85c2951-14fc-4025-99b5-d78d7794cd0c\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " Nov 03 00:31:17.759626 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759580 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-certs" pod "57107b1c-15e3-448e-9dee-de481f4d4bca" (UID: "57107b1c-15e3-448e-9dee-de481f4d4bca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.759626 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759597 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57107b1c-15e3-448e-9dee-de481f4d4bca-istiod-ca-cert" pod "57107b1c-15e3-448e-9dee-de481f4d4bca" (UID: "57107b1c-15e3-448e-9dee-de481f4d4bca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:17.759626 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759623 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-envoy\") pod \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " Nov 03 00:31:17.759962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759664 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-podinfo\") pod \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " Nov 03 00:31:17.759962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759714 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-credential-socket\") pod \"d85c2951-14fc-4025-99b5-d78d7794cd0c\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " Nov 03 00:31:17.759962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759723 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-certs" pod "d85c2951-14fc-4025-99b5-d78d7794cd0c" (UID: "d85c2951-14fc-4025-99b5-d78d7794cd0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.759962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759764 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-token\" (UniqueName: \"kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-token\") pod \"57107b1c-15e3-448e-9dee-de481f4d4bca\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " Nov 03 00:31:17.759962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759793 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-credential-socket" pod "57107b1c-15e3-448e-9dee-de481f4d4bca" (UID: "57107b1c-15e3-448e-9dee-de481f4d4bca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.759962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759813 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-podinfo\") pod \"57107b1c-15e3-448e-9dee-de481f4d4bca\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " Nov 03 00:31:17.759962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759875 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-kube-api-access-h7frv\" (UniqueName: \"kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-kube-api-access-h7frv\") pod \"d85c2951-14fc-4025-99b5-d78d7794cd0c\" (UID: \"d85c2951-14fc-4025-99b5-d78d7794cd0c\") " Nov 03 00:31:17.759962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759910 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-envoy\") pod \"57107b1c-15e3-448e-9dee-de481f4d4bca\" (UID: \"57107b1c-15e3-448e-9dee-de481f4d4bca\") " Nov 03 00:31:17.759962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759913 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-socket" pod "46f7ad4c-9382-4007-96ea-06ed4b82a7ca" (UID: "46f7ad4c-9382-4007-96ea-06ed4b82a7ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.759962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759932 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-data" pod "46f7ad4c-9382-4007-96ea-06ed4b82a7ca" (UID: "46f7ad4c-9382-4007-96ea-06ed4b82a7ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.759962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759936 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-token\" (UniqueName: \"kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-token\") pod \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\" (UID: \"46f7ad4c-9382-4007-96ea-06ed4b82a7ca\") " Nov 03 00:31:17.759962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759948 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-certs" pod "46f7ad4c-9382-4007-96ea-06ed4b82a7ca" (UID: "46f7ad4c-9382-4007-96ea-06ed4b82a7ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.759962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.759962 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-credential-socket" pod "46f7ad4c-9382-4007-96ea-06ed4b82a7ca" (UID: "46f7ad4c-9382-4007-96ea-06ed4b82a7ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.760580 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.760319 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-data" pod "57107b1c-15e3-448e-9dee-de481f4d4bca" (UID: "57107b1c-15e3-448e-9dee-de481f4d4bca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.760580 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.760484 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-data" pod "d85c2951-14fc-4025-99b5-d78d7794cd0c" (UID: "d85c2951-14fc-4025-99b5-d78d7794cd0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.760580 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.760557 6400 scope.go:122] "RemoveContainer" containerID="64ab45fab60f0318861a1f5ac72780be7598778e51d6299c5738b76a51d86b3a" Nov 03 00:31:17.761104 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761074 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761108 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761122 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761135 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761148 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85c2951-14fc-4025-99b5-d78d7794cd0c-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761161 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761175 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57107b1c-15e3-448e-9dee-de481f4d4bca-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761188 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761201 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761213 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761226 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761241 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761255 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761267 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:31:17.761065 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ab45fab60f0318861a1f5ac72780be7598778e51d6299c5738b76a51d86b3a\": container with ID starting with 64ab45fab60f0318861a1f5ac72780be7598778e51d6299c5738b76a51d86b3a not found: ID does not exist" containerID="64ab45fab60f0318861a1f5ac72780be7598778e51d6299c5738b76a51d86b3a" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761297 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ab45fab60f0318861a1f5ac72780be7598778e51d6299c5738b76a51d86b3a"} err="failed to get container status \"64ab45fab60f0318861a1f5ac72780be7598778e51d6299c5738b76a51d86b3a\": rpc error: code = NotFound desc = could not find container \"64ab45fab60f0318861a1f5ac72780be7598778e51d6299c5738b76a51d86b3a\": container with ID starting with 64ab45fab60f0318861a1f5ac72780be7598778e51d6299c5738b76a51d86b3a not found: ID does not exist" Nov 03 00:31:17.761358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761320 6400 scope.go:122] "RemoveContainer" containerID="b1a09acd73a9ecdb949bb69282d1fd01f2ad2a2ca37dc4a075dd2886a7b5a70d" Nov 03 00:31:17.762224 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.761594 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-credential-socket" pod "d85c2951-14fc-4025-99b5-d78d7794cd0c" (UID: "d85c2951-14fc-4025-99b5-d78d7794cd0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.762758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.762671 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-token" pod "d85c2951-14fc-4025-99b5-d78d7794cd0c" (UID: "d85c2951-14fc-4025-99b5-d78d7794cd0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:17.763287 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.763186 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-kube-api-access-4nlj2" pod "46f7ad4c-9382-4007-96ea-06ed4b82a7ca" (UID: "46f7ad4c-9382-4007-96ea-06ed4b82a7ca"). InnerVolumeSpecName "kube-api-access-4nlj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:17.764290 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.764259 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-envoy" pod "46f7ad4c-9382-4007-96ea-06ed4b82a7ca" (UID: "46f7ad4c-9382-4007-96ea-06ed4b82a7ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.764290 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.764280 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-token" pod "57107b1c-15e3-448e-9dee-de481f4d4bca" (UID: "57107b1c-15e3-448e-9dee-de481f4d4bca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:17.764802 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.764778 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-podinfo" pod "57107b1c-15e3-448e-9dee-de481f4d4bca" (UID: "57107b1c-15e3-448e-9dee-de481f4d4bca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:17.764999 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.764937 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-podinfo" pod "d85c2951-14fc-4025-99b5-d78d7794cd0c" (UID: "d85c2951-14fc-4025-99b5-d78d7794cd0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:17.764999 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.764951 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-token" pod "46f7ad4c-9382-4007-96ea-06ed4b82a7ca" (UID: "46f7ad4c-9382-4007-96ea-06ed4b82a7ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:17.765233 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.765118 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-envoy" pod "d85c2951-14fc-4025-99b5-d78d7794cd0c" (UID: "d85c2951-14fc-4025-99b5-d78d7794cd0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.765233 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.765187 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-kube-api-access-h7frv" pod "d85c2951-14fc-4025-99b5-d78d7794cd0c" (UID: "d85c2951-14fc-4025-99b5-d78d7794cd0c"). InnerVolumeSpecName "kube-api-access-h7frv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:17.765712 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.765694 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-envoy" pod "57107b1c-15e3-448e-9dee-de481f4d4bca" (UID: "57107b1c-15e3-448e-9dee-de481f4d4bca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:17.766055 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.766040 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-podinfo" pod "46f7ad4c-9382-4007-96ea-06ed4b82a7ca" (UID: "46f7ad4c-9382-4007-96ea-06ed4b82a7ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:17.766247 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.766232 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-kube-api-access-2bnfg" pod "57107b1c-15e3-448e-9dee-de481f4d4bca" (UID: "57107b1c-15e3-448e-9dee-de481f4d4bca"). InnerVolumeSpecName "kube-api-access-2bnfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:17.771057 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.771042 6400 scope.go:122] "RemoveContainer" containerID="b1a09acd73a9ecdb949bb69282d1fd01f2ad2a2ca37dc4a075dd2886a7b5a70d" Nov 03 00:31:17.771320 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:31:17.771303 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a09acd73a9ecdb949bb69282d1fd01f2ad2a2ca37dc4a075dd2886a7b5a70d\": container with ID starting with b1a09acd73a9ecdb949bb69282d1fd01f2ad2a2ca37dc4a075dd2886a7b5a70d not found: ID does not exist" containerID="b1a09acd73a9ecdb949bb69282d1fd01f2ad2a2ca37dc4a075dd2886a7b5a70d" Nov 03 00:31:17.771374 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.771330 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a09acd73a9ecdb949bb69282d1fd01f2ad2a2ca37dc4a075dd2886a7b5a70d"} err="failed to get container status \"b1a09acd73a9ecdb949bb69282d1fd01f2ad2a2ca37dc4a075dd2886a7b5a70d\": rpc error: code = NotFound desc = could not find container \"b1a09acd73a9ecdb949bb69282d1fd01f2ad2a2ca37dc4a075dd2886a7b5a70d\": container with ID starting with b1a09acd73a9ecdb949bb69282d1fd01f2ad2a2ca37dc4a075dd2886a7b5a70d not found: ID does not exist" Nov 03 00:31:17.771374 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.771346 6400 scope.go:122] "RemoveContainer" containerID="69e6c156d02e4c0fabfb7ed84ec0cbca50e9efd11558419e874998a4288f0fc8" Nov 03 00:31:17.779310 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.779292 6400 scope.go:122] "RemoveContainer" containerID="69e6c156d02e4c0fabfb7ed84ec0cbca50e9efd11558419e874998a4288f0fc8" Nov 03 00:31:17.779571 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:31:17.779554 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e6c156d02e4c0fabfb7ed84ec0cbca50e9efd11558419e874998a4288f0fc8\": container with ID starting with 69e6c156d02e4c0fabfb7ed84ec0cbca50e9efd11558419e874998a4288f0fc8 not found: ID does not exist" containerID="69e6c156d02e4c0fabfb7ed84ec0cbca50e9efd11558419e874998a4288f0fc8" Nov 03 00:31:17.779614 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.779579 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e6c156d02e4c0fabfb7ed84ec0cbca50e9efd11558419e874998a4288f0fc8"} err="failed to get container status \"69e6c156d02e4c0fabfb7ed84ec0cbca50e9efd11558419e874998a4288f0fc8\": rpc error: code = NotFound desc = could not find container \"69e6c156d02e4c0fabfb7ed84ec0cbca50e9efd11558419e874998a4288f0fc8\": container with ID starting with 69e6c156d02e4c0fabfb7ed84ec0cbca50e9efd11558419e874998a4288f0fc8 not found: ID does not exist" Nov 03 00:31:17.862940 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.862878 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.862940 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.862898 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.862940 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.862909 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7frv\" (UniqueName: \"kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-kube-api-access-h7frv\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.862940 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.862919 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57107b1c-15e3-448e-9dee-de481f4d4bca-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.862940 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.862929 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.862940 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.862937 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.863195 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.862946 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bnfg\" (UniqueName: \"kubernetes.io/projected/57107b1c-15e3-448e-9dee-de481f4d4bca-kube-api-access-2bnfg\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.863195 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.862955 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.863195 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.862963 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nlj2\" (UniqueName: \"kubernetes.io/projected/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-kube-api-access-4nlj2\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.863195 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.862972 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85c2951-14fc-4025-99b5-d78d7794cd0c-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.863195 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.862980 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.863195 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.862987 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7ad4c-9382-4007-96ea-06ed4b82a7ca-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:17.863195 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:17.862995 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c2951-14fc-4025-99b5-d78d7794cd0c-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:18.200975 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:18.200898 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp"] Nov 03 00:31:18.205435 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:18.205409 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-rf9vp"] Nov 03 00:31:18.314162 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:18.314134 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp"] Nov 03 00:31:18.319231 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:18.319210 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-qlzdp"] Nov 03 00:31:18.452896 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:18.452666 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc"] Nov 03 00:31:18.475846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:18.469790 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-xx6bc"] Nov 03 00:31:18.541845 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:18.540839 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:19.180018 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:19.179990 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" path="/var/lib/kubelet/pods/46f7ad4c-9382-4007-96ea-06ed4b82a7ca/volumes" Nov 03 00:31:19.180398 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:19.180384 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57107b1c-15e3-448e-9dee-de481f4d4bca" path="/var/lib/kubelet/pods/57107b1c-15e3-448e-9dee-de481f4d4bca/volumes" Nov 03 00:31:19.180767 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:19.180746 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85c2951-14fc-4025-99b5-d78d7794cd0c" path="/var/lib/kubelet/pods/d85c2951-14fc-4025-99b5-d78d7794cd0c/volumes" Nov 03 00:31:20.102297 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:20.102255 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:20.232593 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:20.232564 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:31:20.315809 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:20.315774 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:21.725366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:21.725329 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:23.591050 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:23.591022 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:23.769926 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:23.764535 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:24.521245 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:24.521215 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:26.630119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:26.630089 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:27.151683 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:31:27.151651 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:31:28.677578 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:28.677545 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8"] Nov 03 00:31:28.678051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:28.677755 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" podUID="b261c014-5911-4da3-8ba9-6ed91a665c0d" containerName="istio-proxy" containerID="cri-o://79e7cc57ce89f39246a1c05ff70b3c88071fabc6bbab82e9a1a10796aa21703d" gracePeriod=30 Nov 03 00:31:28.727075 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:28.727045 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b"] Nov 03 00:31:28.727253 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:28.727233 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" podUID="f9c6e0c4-cc76-4dff-8073-8c399e461426" containerName="istio-proxy" containerID="cri-o://7d0ee12727be3e88474b8566d217079ac7842502bf70d21e7163458232f9a9d7" gracePeriod=30 Nov 03 00:31:31.299855 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:31.299753 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:33.870663 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:33.870630 6400 generic.go:356] "Generic (PLEG): container finished" podID="f9c6e0c4-cc76-4dff-8073-8c399e461426" containerID="7d0ee12727be3e88474b8566d217079ac7842502bf70d21e7163458232f9a9d7" exitCode=0 Nov 03 00:31:33.871087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:33.870699 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" event={"ID":"f9c6e0c4-cc76-4dff-8073-8c399e461426","Type":"ContainerDied","Data":"7d0ee12727be3e88474b8566d217079ac7842502bf70d21e7163458232f9a9d7"} Nov 03 00:31:33.872279 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:33.872257 6400 generic.go:356] "Generic (PLEG): container finished" podID="b261c014-5911-4da3-8ba9-6ed91a665c0d" containerID="79e7cc57ce89f39246a1c05ff70b3c88071fabc6bbab82e9a1a10796aa21703d" exitCode=0 Nov 03 00:31:33.872398 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:33.872299 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" event={"ID":"b261c014-5911-4da3-8ba9-6ed91a665c0d","Type":"ContainerDied","Data":"79e7cc57ce89f39246a1c05ff70b3c88071fabc6bbab82e9a1a10796aa21703d"} Nov 03 00:31:34.011178 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.011150 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:31:34.034389 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.034366 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:31:34.104457 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104424 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-credential-socket\") pod \"f9c6e0c4-cc76-4dff-8073-8c399e461426\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " Nov 03 00:31:34.104457 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104469 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-credential-socket\") pod \"b261c014-5911-4da3-8ba9-6ed91a665c0d\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " Nov 03 00:31:34.104711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104487 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-podinfo\") pod \"b261c014-5911-4da3-8ba9-6ed91a665c0d\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " Nov 03 00:31:34.104711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104513 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-data\") pod \"f9c6e0c4-cc76-4dff-8073-8c399e461426\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " Nov 03 00:31:34.104711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104539 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-certs\") pod \"b261c014-5911-4da3-8ba9-6ed91a665c0d\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " Nov 03 00:31:34.104711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104627 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-certs\") pod \"f9c6e0c4-cc76-4dff-8073-8c399e461426\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " Nov 03 00:31:34.104711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104658 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-podinfo\") pod \"f9c6e0c4-cc76-4dff-8073-8c399e461426\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " Nov 03 00:31:34.104711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104685 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-token\" (UniqueName: \"kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-token\") pod \"f9c6e0c4-cc76-4dff-8073-8c399e461426\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " Nov 03 00:31:34.105072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104734 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-data" pod "f9c6e0c4-cc76-4dff-8073-8c399e461426" (UID: "f9c6e0c4-cc76-4dff-8073-8c399e461426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.105072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104742 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b261c014-5911-4da3-8ba9-6ed91a665c0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b261c014-5911-4da3-8ba9-6ed91a665c0d-istiod-ca-cert\") pod \"b261c014-5911-4da3-8ba9-6ed91a665c0d\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " Nov 03 00:31:34.105072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104779 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-credential-socket" pod "b261c014-5911-4da3-8ba9-6ed91a665c0d" (UID: "b261c014-5911-4da3-8ba9-6ed91a665c0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.105072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104863 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c6e0c4-cc76-4dff-8073-8c399e461426-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c6e0c4-cc76-4dff-8073-8c399e461426-istiod-ca-cert\") pod \"f9c6e0c4-cc76-4dff-8073-8c399e461426\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " Nov 03 00:31:34.105072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104946 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-data\") pod \"b261c014-5911-4da3-8ba9-6ed91a665c0d\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " Nov 03 00:31:34.105072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104953 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-certs" pod "f9c6e0c4-cc76-4dff-8073-8c399e461426" (UID: "f9c6e0c4-cc76-4dff-8073-8c399e461426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.105072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104962 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-certs" pod "b261c014-5911-4da3-8ba9-6ed91a665c0d" (UID: "b261c014-5911-4da3-8ba9-6ed91a665c0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.105072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.104977 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-token\" (UniqueName: \"kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-token\") pod \"b261c014-5911-4da3-8ba9-6ed91a665c0d\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " Nov 03 00:31:34.105072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.105041 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b261c014-5911-4da3-8ba9-6ed91a665c0d-istiod-ca-cert" pod "b261c014-5911-4da3-8ba9-6ed91a665c0d" (UID: "b261c014-5911-4da3-8ba9-6ed91a665c0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:34.105072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.105071 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-envoy\") pod \"b261c014-5911-4da3-8ba9-6ed91a665c0d\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " Nov 03 00:31:34.105582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.105094 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-socket\") pod \"b261c014-5911-4da3-8ba9-6ed91a665c0d\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " Nov 03 00:31:34.105582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.105182 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-data" pod "b261c014-5911-4da3-8ba9-6ed91a665c0d" (UID: "b261c014-5911-4da3-8ba9-6ed91a665c0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.105582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.105195 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-kube-api-access-2599l\" (UniqueName: \"kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-kube-api-access-2599l\") pod \"b261c014-5911-4da3-8ba9-6ed91a665c0d\" (UID: \"b261c014-5911-4da3-8ba9-6ed91a665c0d\") " Nov 03 00:31:34.105582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.105288 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-kube-api-access-vb7mm\" (UniqueName: \"kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-kube-api-access-vb7mm\") pod \"f9c6e0c4-cc76-4dff-8073-8c399e461426\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " Nov 03 00:31:34.105582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.105293 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c6e0c4-cc76-4dff-8073-8c399e461426-istiod-ca-cert" pod "f9c6e0c4-cc76-4dff-8073-8c399e461426" (UID: "f9c6e0c4-cc76-4dff-8073-8c399e461426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:34.105582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.105341 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-socket\") pod \"f9c6e0c4-cc76-4dff-8073-8c399e461426\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " Nov 03 00:31:34.105582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.105379 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-envoy\") pod \"f9c6e0c4-cc76-4dff-8073-8c399e461426\" (UID: \"f9c6e0c4-cc76-4dff-8073-8c399e461426\") " Nov 03 00:31:34.105582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.105436 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-credential-socket" pod "f9c6e0c4-cc76-4dff-8073-8c399e461426" (UID: "f9c6e0c4-cc76-4dff-8073-8c399e461426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.105582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.105523 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-socket" pod "b261c014-5911-4da3-8ba9-6ed91a665c0d" (UID: "b261c014-5911-4da3-8ba9-6ed91a665c0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.106254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.105677 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-socket" pod "f9c6e0c4-cc76-4dff-8073-8c399e461426" (UID: "f9c6e0c4-cc76-4dff-8073-8c399e461426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.106254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.106133 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.106254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.106155 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.106254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.106171 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.106254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.106184 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.106254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.106196 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b261c014-5911-4da3-8ba9-6ed91a665c0d-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.106254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.106209 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c6e0c4-cc76-4dff-8073-8c399e461426-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.106254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.106223 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.106254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.106236 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.106254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.106250 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.106793 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.106264 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.107370 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.107340 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-token" pod "b261c014-5911-4da3-8ba9-6ed91a665c0d" (UID: "b261c014-5911-4da3-8ba9-6ed91a665c0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:34.108273 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.108228 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-podinfo" pod "b261c014-5911-4da3-8ba9-6ed91a665c0d" (UID: "b261c014-5911-4da3-8ba9-6ed91a665c0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:34.108380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.108314 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-envoy" pod "b261c014-5911-4da3-8ba9-6ed91a665c0d" (UID: "b261c014-5911-4da3-8ba9-6ed91a665c0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.108380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.108335 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-kube-api-access-vb7mm" pod "f9c6e0c4-cc76-4dff-8073-8c399e461426" (UID: "f9c6e0c4-cc76-4dff-8073-8c399e461426"). InnerVolumeSpecName "kube-api-access-vb7mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:34.108495 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.108391 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-kube-api-access-2599l" pod "b261c014-5911-4da3-8ba9-6ed91a665c0d" (UID: "b261c014-5911-4da3-8ba9-6ed91a665c0d"). InnerVolumeSpecName "kube-api-access-2599l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:34.108561 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.108541 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-token" pod "f9c6e0c4-cc76-4dff-8073-8c399e461426" (UID: "f9c6e0c4-cc76-4dff-8073-8c399e461426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:34.109017 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.108994 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-envoy" pod "f9c6e0c4-cc76-4dff-8073-8c399e461426" (UID: "f9c6e0c4-cc76-4dff-8073-8c399e461426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.109079 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.109021 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-podinfo" pod "f9c6e0c4-cc76-4dff-8073-8c399e461426" (UID: "f9c6e0c4-cc76-4dff-8073-8c399e461426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:34.208005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.207920 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2599l\" (UniqueName: \"kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-kube-api-access-2599l\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.208005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.207957 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb7mm\" (UniqueName: \"kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-kube-api-access-vb7mm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.208005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.207973 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.208005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.207988 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.208005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.208003 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.208315 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.208018 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c6e0c4-cc76-4dff-8073-8c399e461426-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.208315 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.208032 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.208315 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.208045 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b261c014-5911-4da3-8ba9-6ed91a665c0d-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.879678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.879650 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" Nov 03 00:31:34.879678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.879666 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b" event={"ID":"f9c6e0c4-cc76-4dff-8073-8c399e461426","Type":"ContainerDied","Data":"8ce289b8bc5aa625e39ecaf1865826957e6dc9bb1c45fc2c0189b4d9580a186f"} Nov 03 00:31:34.880258 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.879729 6400 scope.go:122] "RemoveContainer" containerID="7d0ee12727be3e88474b8566d217079ac7842502bf70d21e7163458232f9a9d7" Nov 03 00:31:34.881154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.881133 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" event={"ID":"b261c014-5911-4da3-8ba9-6ed91a665c0d","Type":"ContainerDied","Data":"06dbdb51caac7469f308b77ad90ceb7b3c763d8e2414fbe22f3095846263f3dd"} Nov 03 00:31:34.881240 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.881207 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8" Nov 03 00:31:34.889430 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:34.889414 6400 scope.go:122] "RemoveContainer" containerID="79e7cc57ce89f39246a1c05ff70b3c88071fabc6bbab82e9a1a10796aa21703d" Nov 03 00:31:35.094377 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.094338 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b"] Nov 03 00:31:35.143629 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.143599 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-6ww4b"] Nov 03 00:31:35.179396 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.179360 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c6e0c4-cc76-4dff-8073-8c399e461426" path="/var/lib/kubelet/pods/f9c6e0c4-cc76-4dff-8073-8c399e461426/volumes" Nov 03 00:31:35.205874 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.205835 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8"] Nov 03 00:31:35.209625 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.209603 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-qbtr8"] Nov 03 00:31:35.631047 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631018 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2"] Nov 03 00:31:35.631489 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631476 6400 cpu_manager.go:414] "containerMap: removing container" podUID="d85c2951-14fc-4025-99b5-d78d7794cd0c" containerName="istio-proxy" Nov 03 00:31:35.631550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631490 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85c2951-14fc-4025-99b5-d78d7794cd0c" containerName="istio-proxy" Nov 03 00:31:35.631550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631513 6400 cpu_manager.go:414] "containerMap: removing container" podUID="b261c014-5911-4da3-8ba9-6ed91a665c0d" containerName="istio-proxy" Nov 03 00:31:35.631550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631518 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b261c014-5911-4da3-8ba9-6ed91a665c0d" containerName="istio-proxy" Nov 03 00:31:35.631550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631528 6400 cpu_manager.go:414] "containerMap: removing container" podUID="23271791-3a3a-4695-aa8d-33362c295435" containerName="istio-proxy" Nov 03 00:31:35.631550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631532 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23271791-3a3a-4695-aa8d-33362c295435" containerName="istio-proxy" Nov 03 00:31:35.631550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631541 6400 cpu_manager.go:414] "containerMap: removing container" podUID="19a02d3c-eaf0-4e95-991c-aa684c598248" containerName="cleanup" Nov 03 00:31:35.631550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631545 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a02d3c-eaf0-4e95-991c-aa684c598248" containerName="cleanup" Nov 03 00:31:35.631550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631550 6400 cpu_manager.go:414] "containerMap: removing container" podUID="19a02d3c-eaf0-4e95-991c-aa684c598248" containerName="cleanup" Nov 03 00:31:35.631550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631555 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a02d3c-eaf0-4e95-991c-aa684c598248" containerName="cleanup" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631565 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e624205d-ca8e-40eb-b1bd-135005deab0d" containerName="cleanup" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631569 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e624205d-ca8e-40eb-b1bd-135005deab0d" containerName="cleanup" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631589 6400 cpu_manager.go:414] "containerMap: removing container" podUID="7094e012-a61d-4e4b-b8db-f42498a0e762" containerName="maas-api" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631593 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7094e012-a61d-4e4b-b8db-f42498a0e762" containerName="maas-api" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631600 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f9c6e0c4-cc76-4dff-8073-8c399e461426" containerName="istio-proxy" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631605 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c6e0c4-cc76-4dff-8073-8c399e461426" containerName="istio-proxy" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631610 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e624205d-ca8e-40eb-b1bd-135005deab0d" containerName="cleanup" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631614 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e624205d-ca8e-40eb-b1bd-135005deab0d" containerName="cleanup" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631621 6400 cpu_manager.go:414] "containerMap: removing container" podUID="57107b1c-15e3-448e-9dee-de481f4d4bca" containerName="istio-proxy" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631625 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57107b1c-15e3-448e-9dee-de481f4d4bca" containerName="istio-proxy" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631635 6400 cpu_manager.go:414] "containerMap: removing container" podUID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" containerName="istio-proxy" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631639 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" containerName="istio-proxy" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631645 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e624205d-ca8e-40eb-b1bd-135005deab0d" containerName="cleanup" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631649 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e624205d-ca8e-40eb-b1bd-135005deab0d" containerName="cleanup" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631717 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7094e012-a61d-4e4b-b8db-f42498a0e762" containerName="maas-api" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631724 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a02d3c-eaf0-4e95-991c-aa684c598248" containerName="cleanup" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631730 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c6e0c4-cc76-4dff-8073-8c399e461426" containerName="istio-proxy" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631736 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="23271791-3a3a-4695-aa8d-33362c295435" containerName="istio-proxy" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631742 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="57107b1c-15e3-448e-9dee-de481f4d4bca" containerName="istio-proxy" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631748 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="b261c014-5911-4da3-8ba9-6ed91a665c0d" containerName="istio-proxy" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631755 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f7ad4c-9382-4007-96ea-06ed4b82a7ca" containerName="istio-proxy" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631761 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a02d3c-eaf0-4e95-991c-aa684c598248" containerName="cleanup" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631766 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e624205d-ca8e-40eb-b1bd-135005deab0d" containerName="cleanup" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631772 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e624205d-ca8e-40eb-b1bd-135005deab0d" containerName="cleanup" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631776 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e624205d-ca8e-40eb-b1bd-135005deab0d" containerName="cleanup" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631780 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a02d3c-eaf0-4e95-991c-aa684c598248" containerName="cleanup" Nov 03 00:31:35.631986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.631786 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85c2951-14fc-4025-99b5-d78d7794cd0c" containerName="istio-proxy" Nov 03 00:31:35.636627 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.636609 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:35.689265 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.689239 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cb65f3a5-serving-cert\"" Nov 03 00:31:35.691188 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.691161 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2"] Nov 03 00:31:35.726636 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.726613 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrxc7\" (UniqueName: \"kubernetes.io/projected/6daa92b6-9147-47f8-9cb2-c09b81d75359-kube-api-access-xrxc7\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:35.726764 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.726685 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-maas-api-tls\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:35.726764 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.726754 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-metrics-certs\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:35.726887 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.726783 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6daa92b6-9147-47f8-9cb2-c09b81d75359-openshift-service-ca\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:35.827755 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.827717 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-maas-api-tls\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:35.827755 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.827759 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-metrics-certs\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:35.827979 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.827797 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6daa92b6-9147-47f8-9cb2-c09b81d75359-openshift-service-ca\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:35.827979 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:31:35.827905 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-serving-cert: secret "maas-api-e2e-mt-cb65f3a5-serving-cert" not found Nov 03 00:31:35.827979 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:31:35.827971 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-maas-api-tls podName:6daa92b6-9147-47f8-9cb2-c09b81d75359 nodeName:}" failed. No retries permitted until 2026-11-03 00:31:36.327952907 +0000 UTC m=+1543.278893044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-maas-api-tls") pod "maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" (UID: "6daa92b6-9147-47f8-9cb2-c09b81d75359") : secret "maas-api-e2e-mt-cb65f3a5-serving-cert" not found Nov 03 00:31:35.828087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.828019 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrxc7\" (UniqueName: \"kubernetes.io/projected/6daa92b6-9147-47f8-9cb2-c09b81d75359-kube-api-access-xrxc7\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:35.828531 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.828510 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6daa92b6-9147-47f8-9cb2-c09b81d75359-openshift-service-ca\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:35.830484 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.830468 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-metrics-certs\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:35.858913 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.858885 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:35.906610 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:35.906497 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrxc7\" (UniqueName: \"kubernetes.io/projected/6daa92b6-9147-47f8-9cb2-c09b81d75359-kube-api-access-xrxc7\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:36.015469 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:36.015438 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:36.334483 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:36.334444 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-maas-api-tls\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:36.336962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:36.336931 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-maas-api-tls\") pod \"maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:36.346144 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:36.346114 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:36.550025 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:36.549994 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:36.922455 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:36.922434 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2"] Nov 03 00:31:36.924223 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:31:36.924192 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6daa92b6_9147_47f8_9cb2_c09b81d75359.slice/crio-13f0af58d024197dbb7e0643e9599febb45c11e08b029936e40e9aeac61ad408 WatchSource:0}: Error finding container 13f0af58d024197dbb7e0643e9599febb45c11e08b029936e40e9aeac61ad408: Status 404 returned error can't find the container with id 13f0af58d024197dbb7e0643e9599febb45c11e08b029936e40e9aeac61ad408 Nov 03 00:31:36.925682 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:36.925658 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:31:37.185273 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:37.185196 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b261c014-5911-4da3-8ba9-6ed91a665c0d" path="/var/lib/kubelet/pods/b261c014-5911-4da3-8ba9-6ed91a665c0d/volumes" Nov 03 00:31:37.937241 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:37.937205 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" event={"ID":"6daa92b6-9147-47f8-9cb2-c09b81d75359","Type":"ContainerStarted","Data":"caf0069b84e383e25c5a7a84b5b2f1e830bdfa412c8fcfba8a3cb7af89221390"} Nov 03 00:31:37.937663 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:37.937267 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" event={"ID":"6daa92b6-9147-47f8-9cb2-c09b81d75359","Type":"ContainerStarted","Data":"13f0af58d024197dbb7e0643e9599febb45c11e08b029936e40e9aeac61ad408"} Nov 03 00:31:37.937663 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:37.937417 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:38.056272 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:38.056222 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" podStartSLOduration=2.226840631 podStartE2EDuration="3.056207686s" totalImagesPullingTime="829.367055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:38.055063506 +0000 UTC m=+1545.006003676" watchObservedRunningTime="2026-11-03 00:31:38.056207686 +0000 UTC m=+1545.007147843" Nov 03 00:31:39.989867 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:39.989809 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d"] Nov 03 00:31:39.990290 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:39.990057 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" podUID="3080ba7d-614b-4e1b-9d97-a8432e673152" containerName="maas-api" containerID="cri-o://8206ea1fb6bfe54b2bedd683d2fad66bd80139af8023303b850eb4583f5dc188" gracePeriod=30 Nov 03 00:31:40.112011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.111981 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:40.286853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.286420 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:40.338523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.338497 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:31:40.339076 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.339060 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-maas-api-tls\") pod \"3080ba7d-614b-4e1b-9d97-a8432e673152\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " Nov 03 00:31:40.339148 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.339133 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-metrics-certs\") pod \"3080ba7d-614b-4e1b-9d97-a8432e673152\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " Nov 03 00:31:40.339189 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.339165 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3080ba7d-614b-4e1b-9d97-a8432e673152-kube-api-access-zr47k\" (UniqueName: \"kubernetes.io/projected/3080ba7d-614b-4e1b-9d97-a8432e673152-kube-api-access-zr47k\") pod \"3080ba7d-614b-4e1b-9d97-a8432e673152\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " Nov 03 00:31:40.339269 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.339255 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3080ba7d-614b-4e1b-9d97-a8432e673152-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3080ba7d-614b-4e1b-9d97-a8432e673152-openshift-service-ca\") pod \"3080ba7d-614b-4e1b-9d97-a8432e673152\" (UID: \"3080ba7d-614b-4e1b-9d97-a8432e673152\") " Nov 03 00:31:40.339961 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.339741 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3080ba7d-614b-4e1b-9d97-a8432e673152-openshift-service-ca" pod "3080ba7d-614b-4e1b-9d97-a8432e673152" (UID: "3080ba7d-614b-4e1b-9d97-a8432e673152"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:40.339961 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.339937 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3080ba7d-614b-4e1b-9d97-a8432e673152-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:40.341483 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.341463 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-metrics-certs" pod "3080ba7d-614b-4e1b-9d97-a8432e673152" (UID: "3080ba7d-614b-4e1b-9d97-a8432e673152"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:31:40.341867 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.341846 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-maas-api-tls" pod "3080ba7d-614b-4e1b-9d97-a8432e673152" (UID: "3080ba7d-614b-4e1b-9d97-a8432e673152"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:31:40.342022 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.341953 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3080ba7d-614b-4e1b-9d97-a8432e673152-kube-api-access-zr47k" pod "3080ba7d-614b-4e1b-9d97-a8432e673152" (UID: "3080ba7d-614b-4e1b-9d97-a8432e673152"). InnerVolumeSpecName "kube-api-access-zr47k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:40.441437 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.441389 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:40.441437 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.441434 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3080ba7d-614b-4e1b-9d97-a8432e673152-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:40.441679 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.441450 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr47k\" (UniqueName: \"kubernetes.io/projected/3080ba7d-614b-4e1b-9d97-a8432e673152-kube-api-access-zr47k\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:40.993506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.993465 6400 generic.go:356] "Generic (PLEG): container finished" podID="3080ba7d-614b-4e1b-9d97-a8432e673152" containerID="8206ea1fb6bfe54b2bedd683d2fad66bd80139af8023303b850eb4583f5dc188" exitCode=0 Nov 03 00:31:40.994019 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.993568 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" event={"ID":"3080ba7d-614b-4e1b-9d97-a8432e673152","Type":"ContainerDied","Data":"8206ea1fb6bfe54b2bedd683d2fad66bd80139af8023303b850eb4583f5dc188"} Nov 03 00:31:40.994019 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.993624 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" event={"ID":"3080ba7d-614b-4e1b-9d97-a8432e673152","Type":"ContainerDied","Data":"c3b9af5727590c05630eb02f4b7d84794767dade770355fda71b503296e734c9"} Nov 03 00:31:40.994019 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.993656 6400 scope.go:122] "RemoveContainer" containerID="8206ea1fb6bfe54b2bedd683d2fad66bd80139af8023303b850eb4583f5dc188" Nov 03 00:31:40.994019 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:40.993846 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d" Nov 03 00:31:41.022231 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:41.022211 6400 scope.go:122] "RemoveContainer" containerID="8206ea1fb6bfe54b2bedd683d2fad66bd80139af8023303b850eb4583f5dc188" Nov 03 00:31:41.022576 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:31:41.022551 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8206ea1fb6bfe54b2bedd683d2fad66bd80139af8023303b850eb4583f5dc188\": container with ID starting with 8206ea1fb6bfe54b2bedd683d2fad66bd80139af8023303b850eb4583f5dc188 not found: ID does not exist" containerID="8206ea1fb6bfe54b2bedd683d2fad66bd80139af8023303b850eb4583f5dc188" Nov 03 00:31:41.022665 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:41.022589 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8206ea1fb6bfe54b2bedd683d2fad66bd80139af8023303b850eb4583f5dc188"} err="failed to get container status \"8206ea1fb6bfe54b2bedd683d2fad66bd80139af8023303b850eb4583f5dc188\": rpc error: code = NotFound desc = could not find container \"8206ea1fb6bfe54b2bedd683d2fad66bd80139af8023303b850eb4583f5dc188\": container with ID starting with 8206ea1fb6bfe54b2bedd683d2fad66bd80139af8023303b850eb4583f5dc188 not found: ID does not exist" Nov 03 00:31:41.156546 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:41.156508 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d"] Nov 03 00:31:41.160095 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:41.160070 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-be578d-654b6bc958-6mh4d"] Nov 03 00:31:41.181036 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:41.181009 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3080ba7d-614b-4e1b-9d97-a8432e673152" path="/var/lib/kubelet/pods/3080ba7d-614b-4e1b-9d97-a8432e673152/volumes" Nov 03 00:31:41.826786 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:41.826753 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:43.948703 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:43.948674 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:31:46.862736 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:46.862697 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:47.804289 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:47.804252 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:48.029517 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:48.029484 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:49.329090 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:49.329052 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:49.860696 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:49.860659 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:50.480938 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:50.480904 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:50.679545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:50.679513 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:51.103521 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:51.103484 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:53.469719 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:53.469685 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:31:53.921509 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:53.921478 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:56.487103 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:56.487053 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:57.153081 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:31:57.153045 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:31:57.696767 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:57.696719 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:57.887284 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:57.887246 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:58.911414 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:31:58.911382 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:02.073401 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:02.073361 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:03.742096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:03.742061 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:03.913358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:03.913321 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:04.077980 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:04.077946 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:07.290407 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:07.290366 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:08.665008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:08.664975 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:10.995059 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:10.995028 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq"] Nov 03 00:32:10.995505 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:10.995492 6400 cpu_manager.go:414] "containerMap: removing container" podUID="19a02d3c-eaf0-4e95-991c-aa684c598248" containerName="cleanup" Nov 03 00:32:10.995556 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:10.995506 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a02d3c-eaf0-4e95-991c-aa684c598248" containerName="cleanup" Nov 03 00:32:10.995556 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:10.995547 6400 cpu_manager.go:414] "containerMap: removing container" podUID="3080ba7d-614b-4e1b-9d97-a8432e673152" containerName="maas-api" Nov 03 00:32:10.995556 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:10.995552 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3080ba7d-614b-4e1b-9d97-a8432e673152" containerName="maas-api" Nov 03 00:32:10.995645 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:10.995630 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="3080ba7d-614b-4e1b-9d97-a8432e673152" containerName="maas-api" Nov 03 00:32:10.998772 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:10.998755 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.054245 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.054213 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cb65f3a5-openshift-default-dockercfg-fxcdx\"" Nov 03 00:32:11.107857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.107778 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq"] Nov 03 00:32:11.126469 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.126429 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.126469 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.126480 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.126739 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.126503 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.126739 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.126660 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079b7104-4e09-41de-b57d-79c1417f7336-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.126739 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.126715 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.126934 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.126801 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.126934 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.126917 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079b7104-4e09-41de-b57d-79c1417f7336-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.127041 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.127017 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.127106 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.127088 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxdn\" (UniqueName: \"kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-kube-api-access-4fxdn\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.228841 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.228773 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.229034 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.228895 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxdn\" (UniqueName: \"kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-kube-api-access-4fxdn\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.229034 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.228955 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.229034 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.228990 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.229034 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.229014 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.229248 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.229049 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079b7104-4e09-41de-b57d-79c1417f7336-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.229248 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.229075 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.229248 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.229125 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.229248 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.229198 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079b7104-4e09-41de-b57d-79c1417f7336-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.229458 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.229400 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.229458 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.229434 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.229712 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.229685 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.229873 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.229697 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.230286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.230266 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079b7104-4e09-41de-b57d-79c1417f7336-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.231589 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.231569 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.231986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.231966 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079b7104-4e09-41de-b57d-79c1417f7336-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.286661 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.286635 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxdn\" (UniqueName: \"kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-kube-api-access-4fxdn\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.286847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.286644 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.311731 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.311693 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:11.740130 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.740103 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq"] Nov 03 00:32:11.741743 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:32:11.741716 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079b7104_4e09_41de_b57d_79c1417f7336.slice/crio-1230ae517f4280026167ea44866e35a08535d223a9cf6275f36571a828d9105e WatchSource:0}: Error finding container 1230ae517f4280026167ea44866e35a08535d223a9cf6275f36571a828d9105e: Status 404 returned error can't find the container with id 1230ae517f4280026167ea44866e35a08535d223a9cf6275f36571a828d9105e Nov 03 00:32:11.744176 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.744143 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:32:11.744250 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.744211 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:32:11.744250 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:11.744240 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:32:12.225619 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:12.225584 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" event={"ID":"079b7104-4e09-41de-b57d-79c1417f7336","Type":"ContainerStarted","Data":"95e8d6b468bb851467854c4f7928f3aba15386f06886fb98745ece680d7e8340"} Nov 03 00:32:12.226120 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:12.225657 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" event={"ID":"079b7104-4e09-41de-b57d-79c1417f7336","Type":"ContainerStarted","Data":"1230ae517f4280026167ea44866e35a08535d223a9cf6275f36571a828d9105e"} Nov 03 00:32:12.312276 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:12.312242 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:12.313924 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:12.313889 6400 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:32:12.313979 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:12.313948 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" podUID="079b7104-4e09-41de-b57d-79c1417f7336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 00:32:12.346426 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:12.346374 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" podStartSLOduration=2.346362086 podStartE2EDuration="2.346362086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:12.345026527 +0000 UTC m=+1579.295966685" watchObservedRunningTime="2026-11-03 00:32:12.346362086 +0000 UTC m=+1579.297302243" Nov 03 00:32:13.115221 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.115191 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4"] Nov 03 00:32:13.119066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.119042 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:13.154009 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.153974 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-metrics-certs\") pod \"maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:13.154138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.154031 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:13.154138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.154119 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b82a6a0-d03f-4a10-af02-a0770917e943-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:13.154264 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.154196 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv59m\" (UniqueName: \"kubernetes.io/projected/2b82a6a0-d03f-4a10-af02-a0770917e943-kube-api-access-fv59m\") pod \"maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:13.173478 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.173454 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3df441-serving-cert\"" Nov 03 00:32:13.208315 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.208284 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4"] Nov 03 00:32:13.255810 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.255777 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b82a6a0-d03f-4a10-af02-a0770917e943-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:13.255810 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.255811 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv59m\" (UniqueName: \"kubernetes.io/projected/2b82a6a0-d03f-4a10-af02-a0770917e943-kube-api-access-fv59m\") pod \"maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:13.256384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.255989 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-metrics-certs\") pod \"maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:13.256384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.256037 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:13.256384 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:32:13.256182 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-serving-cert: secret "maas-api-e2e-ipp-a-3df441-serving-cert" not found Nov 03 00:32:13.256384 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:32:13.256243 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-maas-api-tls podName:2b82a6a0-d03f-4a10-af02-a0770917e943 nodeName:}" failed. No retries permitted until 2026-11-03 00:32:13.756223108 +0000 UTC m=+1580.707163243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-maas-api-tls") pod "maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" (UID: "2b82a6a0-d03f-4a10-af02-a0770917e943") : secret "maas-api-e2e-ipp-a-3df441-serving-cert" not found Nov 03 00:32:13.256628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.256566 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b82a6a0-d03f-4a10-af02-a0770917e943-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:13.258978 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.258945 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-metrics-certs\") pod \"maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:13.312119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.312086 6400 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:32:13.312263 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.312148 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" podUID="079b7104-4e09-41de-b57d-79c1417f7336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 00:32:13.313057 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.313035 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv59m\" (UniqueName: \"kubernetes.io/projected/2b82a6a0-d03f-4a10-af02-a0770917e943-kube-api-access-fv59m\") pod \"maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:13.427900 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.424436 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:13.591006 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.590873 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:13.755244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.755149 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:13.761727 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.761706 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:13.764156 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:13.764138 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:14.033172 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:14.033149 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:14.312141 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:14.312058 6400 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:32:14.312141 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:14.312107 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" podUID="079b7104-4e09-41de-b57d-79c1417f7336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 00:32:14.446249 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:14.446224 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4"] Nov 03 00:32:14.447001 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:32:14.446964 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b82a6a0_d03f_4a10_af02_a0770917e943.slice/crio-bc5c5ad02224c2291a0a084bd17940b870645081057105a077df77714d620d64 WatchSource:0}: Error finding container bc5c5ad02224c2291a0a084bd17940b870645081057105a077df77714d620d64: Status 404 returned error can't find the container with id bc5c5ad02224c2291a0a084bd17940b870645081057105a077df77714d620d64 Nov 03 00:32:15.249508 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:15.249469 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" event={"ID":"2b82a6a0-d03f-4a10-af02-a0770917e943","Type":"ContainerStarted","Data":"bc5c5ad02224c2291a0a084bd17940b870645081057105a077df77714d620d64"} Nov 03 00:32:15.312979 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:15.312758 6400 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:32:15.312979 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:15.312812 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" podUID="079b7104-4e09-41de-b57d-79c1417f7336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 00:32:16.256690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:16.256657 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" event={"ID":"2b82a6a0-d03f-4a10-af02-a0770917e943","Type":"ContainerStarted","Data":"daa1ea0896aa8112d8fdfa972d4f7d392852560e4e5d0933a02eac693933feee"} Nov 03 00:32:16.256949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:16.256794 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:16.312954 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:16.312924 6400 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:32:16.313100 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:16.312986 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" podUID="079b7104-4e09-41de-b57d-79c1417f7336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 00:32:17.313249 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:17.313212 6400 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:32:17.313857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:17.313280 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" podUID="079b7104-4e09-41de-b57d-79c1417f7336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 00:32:18.313887 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:18.313806 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:18.316420 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:18.316402 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:18.316656 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:18.316636 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:18.317264 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:18.317246 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:32:18.536816 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:18.536763 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" podStartSLOduration=4.713730554 podStartE2EDuration="5.536748486s" totalImagesPullingTime="823.017932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:16.405117802 +0000 UTC m=+1583.356057963" watchObservedRunningTime="2026-11-03 00:32:18.536748486 +0000 UTC m=+1585.487688634" Nov 03 00:32:20.737988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:20.737957 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:32:22.266964 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:22.266939 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:32:23.189703 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:23.189661 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:26.145674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.145642 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64"] Nov 03 00:32:26.149659 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.149642 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.251833 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.251789 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.251976 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.251955 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.252031 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.251981 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.252086 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.252027 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.252174 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.252156 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.252233 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.252193 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.252328 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.252305 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.252372 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.252354 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sql4\" (UniqueName: \"kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-kube-api-access-6sql4\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.252451 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.252437 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.353972 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.353919 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.353972 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.353966 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.354197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.354016 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.354197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.354077 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.354197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.354105 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.354197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.354155 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.354197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.354190 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sql4\" (UniqueName: \"kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-kube-api-access-6sql4\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.355097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.354229 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.355097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.354273 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.355097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.354603 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.355097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.354728 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.355278 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.355123 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64"] Nov 03 00:32:26.355809 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.355784 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.356225 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.356173 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.356360 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.356320 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.357529 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.357506 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.366291 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.366268 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.410937 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.410890 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.411165 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.411149 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sql4\" (UniqueName: \"kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-kube-api-access-6sql4\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.463199 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.463173 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:26.902367 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.902338 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64"] Nov 03 00:32:26.903932 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:32:26.903905 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c136a07_73a1_4f21_b543_e2ff13dc9bac.slice/crio-aea08c9273718c4ba3c799c5705bbc34e797694317bc2ec1619c109105eb7f13 WatchSource:0}: Error finding container aea08c9273718c4ba3c799c5705bbc34e797694317bc2ec1619c109105eb7f13: Status 404 returned error can't find the container with id aea08c9273718c4ba3c799c5705bbc34e797694317bc2ec1619c109105eb7f13 Nov 03 00:32:26.906511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.906480 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:32:26.906590 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.906545 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:32:26.906590 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:26.906575 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:32:27.154667 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:32:27.154584 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:32:27.316896 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:27.316860 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" event={"ID":"7c136a07-73a1-4f21-b543-e2ff13dc9bac","Type":"ContainerStarted","Data":"d4fcb308262ded728f78e73f1327481fcb3579f9572b6c388b36ff8982cdec2b"} Nov 03 00:32:27.317459 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:27.316917 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" event={"ID":"7c136a07-73a1-4f21-b543-e2ff13dc9bac","Type":"ContainerStarted","Data":"aea08c9273718c4ba3c799c5705bbc34e797694317bc2ec1619c109105eb7f13"} Nov 03 00:32:27.464161 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:27.464102 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:27.465636 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:27.465609 6400 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:32:27.465767 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:27.465660 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" podUID="7c136a07-73a1-4f21-b543-e2ff13dc9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:32:27.484333 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:27.484291 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" podStartSLOduration=1.484278468 podStartE2EDuration="1.484278468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:27.481074437 +0000 UTC m=+1594.432014597" watchObservedRunningTime="2026-11-03 00:32:27.484278468 +0000 UTC m=+1594.435218624" Nov 03 00:32:28.166656 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:28.166625 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:28.464050 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:28.463979 6400 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:32:28.464050 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:28.464032 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" podUID="7c136a07-73a1-4f21-b543-e2ff13dc9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:32:28.852841 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:28.852237 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:29.062641 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:29.061541 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:29.229705 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:29.229613 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:29.463665 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:29.463633 6400 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:32:29.463860 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:29.463680 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" podUID="7c136a07-73a1-4f21-b543-e2ff13dc9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:32:30.466251 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:30.466215 6400 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:32:30.467546 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:30.467514 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" podUID="7c136a07-73a1-4f21-b543-e2ff13dc9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:32:31.463703 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:31.463671 6400 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:32:31.463904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:31.463722 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" podUID="7c136a07-73a1-4f21-b543-e2ff13dc9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:32:32.464275 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:32.464241 6400 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:32:32.464654 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:32.464287 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" podUID="7c136a07-73a1-4f21-b543-e2ff13dc9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:32:33.249396 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:33.249367 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:33.464319 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:33.464284 6400 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:32:33.464700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:33.464332 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" podUID="7c136a07-73a1-4f21-b543-e2ff13dc9bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 00:32:34.467992 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:34.467968 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:34.468369 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:34.468351 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:34.469010 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:34.468991 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:32:37.967345 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:37.967307 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:42.755513 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:42.755483 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:48.069728 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:48.069687 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:49.089332 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:49.089304 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:32:53.716629 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:53.716598 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr"] Nov 03 00:32:53.717124 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:53.716882 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" podUID="08e5e225-44ad-49a7-8b30-94ed44e3e344" containerName="maas-api" containerID="cri-o://b0a56b970371910ed276eda9b32931fec8c6ede07304faa1021c005f25131cde" gracePeriod=30 Nov 03 00:32:53.846711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:53.846656 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:54.086652 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.086626 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:32:54.224492 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.224446 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08e5e225-44ad-49a7-8b30-94ed44e3e344-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e5e225-44ad-49a7-8b30-94ed44e3e344-openshift-service-ca\") pod \"08e5e225-44ad-49a7-8b30-94ed44e3e344\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " Nov 03 00:32:54.224678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.224543 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-maas-api-tls\") pod \"08e5e225-44ad-49a7-8b30-94ed44e3e344\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " Nov 03 00:32:54.224678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.224670 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-metrics-certs\") pod \"08e5e225-44ad-49a7-8b30-94ed44e3e344\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " Nov 03 00:32:54.224801 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.224712 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e5e225-44ad-49a7-8b30-94ed44e3e344-kube-api-access-l656q\" (UniqueName: \"kubernetes.io/projected/08e5e225-44ad-49a7-8b30-94ed44e3e344-kube-api-access-l656q\") pod \"08e5e225-44ad-49a7-8b30-94ed44e3e344\" (UID: \"08e5e225-44ad-49a7-8b30-94ed44e3e344\") " Nov 03 00:32:54.225424 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.225395 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e5e225-44ad-49a7-8b30-94ed44e3e344-openshift-service-ca" pod "08e5e225-44ad-49a7-8b30-94ed44e3e344" (UID: "08e5e225-44ad-49a7-8b30-94ed44e3e344"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:32:54.228938 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.228895 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e5e225-44ad-49a7-8b30-94ed44e3e344-kube-api-access-l656q" pod "08e5e225-44ad-49a7-8b30-94ed44e3e344" (UID: "08e5e225-44ad-49a7-8b30-94ed44e3e344"). InnerVolumeSpecName "kube-api-access-l656q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:32:54.234298 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.234266 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-maas-api-tls" pod "08e5e225-44ad-49a7-8b30-94ed44e3e344" (UID: "08e5e225-44ad-49a7-8b30-94ed44e3e344"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:32:54.235335 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.235312 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-metrics-certs" pod "08e5e225-44ad-49a7-8b30-94ed44e3e344" (UID: "08e5e225-44ad-49a7-8b30-94ed44e3e344"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:32:54.342948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.328913 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e5e225-44ad-49a7-8b30-94ed44e3e344-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:54.342948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.328954 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:54.342948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.328969 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08e5e225-44ad-49a7-8b30-94ed44e3e344-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:54.342948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.328984 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l656q\" (UniqueName: \"kubernetes.io/projected/08e5e225-44ad-49a7-8b30-94ed44e3e344-kube-api-access-l656q\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:54.484941 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.484907 6400 generic.go:356] "Generic (PLEG): container finished" podID="08e5e225-44ad-49a7-8b30-94ed44e3e344" containerID="b0a56b970371910ed276eda9b32931fec8c6ede07304faa1021c005f25131cde" exitCode=0 Nov 03 00:32:54.485117 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.484952 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" Nov 03 00:32:54.485117 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.484988 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" event={"ID":"08e5e225-44ad-49a7-8b30-94ed44e3e344","Type":"ContainerDied","Data":"b0a56b970371910ed276eda9b32931fec8c6ede07304faa1021c005f25131cde"} Nov 03 00:32:54.485117 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.485046 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr" event={"ID":"08e5e225-44ad-49a7-8b30-94ed44e3e344","Type":"ContainerDied","Data":"d3b5f4a50a410c4f8f80918a1cd417685ee531259a112edd2c8723a14ad9030b"} Nov 03 00:32:54.485117 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.485072 6400 scope.go:122] "RemoveContainer" containerID="b0a56b970371910ed276eda9b32931fec8c6ede07304faa1021c005f25131cde" Nov 03 00:32:54.495775 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.495759 6400 scope.go:122] "RemoveContainer" containerID="b0a56b970371910ed276eda9b32931fec8c6ede07304faa1021c005f25131cde" Nov 03 00:32:54.496052 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:32:54.496034 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a56b970371910ed276eda9b32931fec8c6ede07304faa1021c005f25131cde\": container with ID starting with b0a56b970371910ed276eda9b32931fec8c6ede07304faa1021c005f25131cde not found: ID does not exist" containerID="b0a56b970371910ed276eda9b32931fec8c6ede07304faa1021c005f25131cde" Nov 03 00:32:54.496095 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.496061 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a56b970371910ed276eda9b32931fec8c6ede07304faa1021c005f25131cde"} err="failed to get container status \"b0a56b970371910ed276eda9b32931fec8c6ede07304faa1021c005f25131cde\": rpc error: code = NotFound desc = could not find container \"b0a56b970371910ed276eda9b32931fec8c6ede07304faa1021c005f25131cde\": container with ID starting with b0a56b970371910ed276eda9b32931fec8c6ede07304faa1021c005f25131cde not found: ID does not exist" Nov 03 00:32:54.648092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.648014 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr"] Nov 03 00:32:54.651324 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:54.651292 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c521ba-7f6446dc48-2hbwr"] Nov 03 00:32:55.179911 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:55.179882 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e5e225-44ad-49a7-8b30-94ed44e3e344" path="/var/lib/kubelet/pods/08e5e225-44ad-49a7-8b30-94ed44e3e344/volumes" Nov 03 00:32:55.216669 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:55.216635 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:55.602388 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:32:55.602358 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:57.153956 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:32:57.153916 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:33:00.150558 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:00.150523 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs"] Nov 03 00:33:00.246139 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:00.246103 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:00.452893 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:00.452713 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:00.799908 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:00.799875 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:00.962369 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:00.962338 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:02.369356 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:02.369323 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:02.881645 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:02.881612 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:03.795316 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:03.795266 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:04.386866 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:04.386801 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:04.806853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:04.806732 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:05.599691 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:05.599658 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:07.617946 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:07.617908 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:08.065870 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:08.065812 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:10.013341 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:10.013307 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:10.239058 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:10.239011 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:10.392382 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:10.392350 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:10.839877 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:10.839771 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:13.901949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:13.901907 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:14.131369 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:14.131328 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:15.171721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.171675 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" podUID="9ebaf2f2-ebc4-4baa-b326-d9c2491e1735" containerName="main" containerID="cri-o://db15702e925fa2159bbe4c345392203de94bf70219b52580ce9595f45f5c97bc" gracePeriod=45 Nov 03 00:33:15.491066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.491035 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:33:15.561913 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.561886 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-home\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-home\") pod \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " Nov 03 00:33:15.562074 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.562040 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-model-cache\") pod \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " Nov 03 00:33:15.562074 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.562062 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tmp-dir\") pod \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " Nov 03 00:33:15.562159 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.562083 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tls-certs\") pod \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " Nov 03 00:33:15.562159 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.562152 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-kube-api-access-5trfq\" (UniqueName: \"kubernetes.io/projected/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-kube-api-access-5trfq\") pod \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " Nov 03 00:33:15.562261 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.562157 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-home" pod "9ebaf2f2-ebc4-4baa-b326-d9c2491e1735" (UID: "9ebaf2f2-ebc4-4baa-b326-d9c2491e1735"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:15.562261 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.562240 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-dshm\") pod \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\" (UID: \"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735\") " Nov 03 00:33:15.562360 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.562329 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tmp-dir" pod "9ebaf2f2-ebc4-4baa-b326-d9c2491e1735" (UID: "9ebaf2f2-ebc4-4baa-b326-d9c2491e1735"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:15.562360 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.562340 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-model-cache" pod "9ebaf2f2-ebc4-4baa-b326-d9c2491e1735" (UID: "9ebaf2f2-ebc4-4baa-b326-d9c2491e1735"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:15.562703 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.562672 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:15.562703 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.562698 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:15.562703 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.562707 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:15.564593 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.564564 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-kube-api-access-5trfq" pod "9ebaf2f2-ebc4-4baa-b326-d9c2491e1735" (UID: "9ebaf2f2-ebc4-4baa-b326-d9c2491e1735"). InnerVolumeSpecName "kube-api-access-5trfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:15.564721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.564663 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tls-certs" pod "9ebaf2f2-ebc4-4baa-b326-d9c2491e1735" (UID: "9ebaf2f2-ebc4-4baa-b326-d9c2491e1735"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:33:15.564850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.564717 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-dshm" pod "9ebaf2f2-ebc4-4baa-b326-d9c2491e1735" (UID: "9ebaf2f2-ebc4-4baa-b326-d9c2491e1735"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:15.622171 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.622131 6400 generic.go:356] "Generic (PLEG): container finished" podID="9ebaf2f2-ebc4-4baa-b326-d9c2491e1735" containerID="db15702e925fa2159bbe4c345392203de94bf70219b52580ce9595f45f5c97bc" exitCode=0 Nov 03 00:33:15.622353 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.622186 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" Nov 03 00:33:15.622353 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.622176 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" event={"ID":"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735","Type":"ContainerDied","Data":"db15702e925fa2159bbe4c345392203de94bf70219b52580ce9595f45f5c97bc"} Nov 03 00:33:15.622353 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.622315 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs" event={"ID":"9ebaf2f2-ebc4-4baa-b326-d9c2491e1735","Type":"ContainerDied","Data":"cbee76efd7eabc020755338b71a5e0ded68ab2e428b8b7225eceb218b4a8da1c"} Nov 03 00:33:15.622492 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.622342 6400 scope.go:122] "RemoveContainer" containerID="db15702e925fa2159bbe4c345392203de94bf70219b52580ce9595f45f5c97bc" Nov 03 00:33:15.635661 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.635639 6400 scope.go:122] "RemoveContainer" containerID="db15702e925fa2159bbe4c345392203de94bf70219b52580ce9595f45f5c97bc" Nov 03 00:33:15.636053 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:33:15.636032 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db15702e925fa2159bbe4c345392203de94bf70219b52580ce9595f45f5c97bc\": container with ID starting with db15702e925fa2159bbe4c345392203de94bf70219b52580ce9595f45f5c97bc not found: ID does not exist" containerID="db15702e925fa2159bbe4c345392203de94bf70219b52580ce9595f45f5c97bc" Nov 03 00:33:15.636152 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.636061 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db15702e925fa2159bbe4c345392203de94bf70219b52580ce9595f45f5c97bc"} err="failed to get container status \"db15702e925fa2159bbe4c345392203de94bf70219b52580ce9595f45f5c97bc\": rpc error: code = NotFound desc = could not find container \"db15702e925fa2159bbe4c345392203de94bf70219b52580ce9595f45f5c97bc\": container with ID starting with db15702e925fa2159bbe4c345392203de94bf70219b52580ce9595f45f5c97bc not found: ID does not exist" Nov 03 00:33:15.663510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.663478 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:15.663510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.663507 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5trfq\" (UniqueName: \"kubernetes.io/projected/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-kube-api-access-5trfq\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:15.663696 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.663524 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:15.798406 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.798371 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs"] Nov 03 00:33:15.814931 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:15.804296 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cb65f3a5/e2e-lifecycle-model-cb65f3a5-kserve-76fd8bf565-5j9gs"] Nov 03 00:33:16.212299 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:16.212208 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:16.375471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:16.375429 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:16.533071 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:16.533029 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:17.179145 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:17.179113 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ebaf2f2-ebc4-4baa-b326-d9c2491e1735" path="/var/lib/kubelet/pods/9ebaf2f2-ebc4-4baa-b326-d9c2491e1735/volumes" Nov 03 00:33:17.377788 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:17.377739 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:17.939540 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:17.939510 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:18.111005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:18.110962 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:20.163936 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:20.163905 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:20.647923 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:20.647734 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:23.207378 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:23.207350 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:27.152382 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:33:27.152344 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:33:27.880662 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:27.880624 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:28.503397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:28.503360 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:32.542941 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:32.542897 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:33.601848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:33.601806 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:33:36.296691 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:36.296652 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:36.762190 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:36.762104 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:37.249281 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.249248 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:37.618309 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.618273 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz"] Nov 03 00:33:37.619055 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.619037 6400 cpu_manager.go:414] "containerMap: removing container" podUID="9ebaf2f2-ebc4-4baa-b326-d9c2491e1735" containerName="main" Nov 03 00:33:37.619106 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.619059 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ebaf2f2-ebc4-4baa-b326-d9c2491e1735" containerName="main" Nov 03 00:33:37.619144 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.619106 6400 cpu_manager.go:414] "containerMap: removing container" podUID="08e5e225-44ad-49a7-8b30-94ed44e3e344" containerName="maas-api" Nov 03 00:33:37.619144 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.619113 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e5e225-44ad-49a7-8b30-94ed44e3e344" containerName="maas-api" Nov 03 00:33:37.619269 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.619257 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e5e225-44ad-49a7-8b30-94ed44e3e344" containerName="maas-api" Nov 03 00:33:37.619306 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.619274 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ebaf2f2-ebc4-4baa-b326-d9c2491e1735" containerName="main" Nov 03 00:33:37.624754 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.624725 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.725660 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.725628 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:37.726684 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.726654 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz"] Nov 03 00:33:37.788695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.788663 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-eb20df-openshift-default-dockercfg-pckqc\"" Nov 03 00:33:37.793922 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.790253 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-podinfo\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.794214 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.794193 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-socket\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.794370 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.794353 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-envoy\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.794493 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.794480 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-token\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.794616 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.794588 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e679802-f6b3-49e5-9c52-f523957ce3ed-istiod-ca-cert\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.794755 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.794739 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4pjq\" (UniqueName: \"kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-kube-api-access-x4pjq\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.794994 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.794957 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-certs\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.795180 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.795135 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-credential-socket\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.796879 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.795372 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-data\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.898376 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.898292 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-token\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.898376 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.898355 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e679802-f6b3-49e5-9c52-f523957ce3ed-istiod-ca-cert\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.898567 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.898412 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pjq\" (UniqueName: \"kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-kube-api-access-x4pjq\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.898567 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.898487 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-certs\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.898567 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.898517 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-credential-socket\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.898719 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.898599 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-data\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.898719 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.898646 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-podinfo\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.898719 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.898713 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-socket\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.898884 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.898763 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-envoy\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.898941 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.898894 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-certs\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.898994 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.898965 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-data\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.899052 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.899032 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-credential-socket\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.899202 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.899170 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-socket\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.899202 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.899193 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e679802-f6b3-49e5-9c52-f523957ce3ed-istiod-ca-cert\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.901593 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.901534 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-podinfo\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.901801 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.901768 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-envoy\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.955442 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.955412 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pjq\" (UniqueName: \"kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-kube-api-access-x4pjq\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.955845 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.955803 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-token\") pod \"e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:37.960787 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:37.960767 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:38.181899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:38.153489 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:38.366847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:38.364052 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:38.411449 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:38.411411 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz"] Nov 03 00:33:38.539246 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:38.518782 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:33:38.539246 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:38.518913 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:33:38.539246 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:38.518964 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:33:38.560407 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:38.560374 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:38.725366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:38.725240 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:38.750898 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:38.749766 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" event={"ID":"1e679802-f6b3-49e5-9c52-f523957ce3ed","Type":"ContainerStarted","Data":"b94b032c32511accf166739d2f73eaa6d6b8ab2e7489de9e4364df3923116859"} Nov 03 00:33:38.750898 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:38.749856 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" event={"ID":"1e679802-f6b3-49e5-9c52-f523957ce3ed","Type":"ContainerStarted","Data":"a9edd3a42c365e548774bd0aa42d1a603de55507de0c8a531063d08bc915425d"} Nov 03 00:33:38.869712 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:38.869652 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" podStartSLOduration=1.869638366 podStartE2EDuration="1.869638366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:38.86670138 +0000 UTC m=+1665.817641549" watchObservedRunningTime="2026-11-03 00:33:38.869638366 +0000 UTC m=+1665.820578522" Nov 03 00:33:38.961760 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:38.961729 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:39.966975 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:39.966941 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:40.761443 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:40.761414 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:40.762383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:40.762364 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:33:41.690053 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:41.690020 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:41.867244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:41.867216 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:42.040884 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:42.040855 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:42.238541 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:42.238514 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:44.086832 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:44.086793 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:46.400947 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:46.400908 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:46.614581 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:46.614553 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:46.727896 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:46.727343 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld"] Nov 03 00:33:46.727896 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:46.727715 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" podUID="87be3afe-a55b-48d2-a890-2788ea461965" containerName="maas-api" containerID="cri-o://1af4a5d9a71ea35ebf47583531c2662ff4cf29f7db6b6003f081bd606192c1c0" gracePeriod=30 Nov 03 00:33:46.837892 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:46.837851 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:47.253403 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.253373 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:33:47.357077 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.357044 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87be3afe-a55b-48d2-a890-2788ea461965-kube-api-access-n82v2\" (UniqueName: \"kubernetes.io/projected/87be3afe-a55b-48d2-a890-2788ea461965-kube-api-access-n82v2\") pod \"87be3afe-a55b-48d2-a890-2788ea461965\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " Nov 03 00:33:47.357277 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.357128 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-maas-api-tls\") pod \"87be3afe-a55b-48d2-a890-2788ea461965\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " Nov 03 00:33:47.357277 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.357164 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87be3afe-a55b-48d2-a890-2788ea461965-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87be3afe-a55b-48d2-a890-2788ea461965-openshift-service-ca\") pod \"87be3afe-a55b-48d2-a890-2788ea461965\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " Nov 03 00:33:47.357277 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.357235 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-metrics-certs\") pod \"87be3afe-a55b-48d2-a890-2788ea461965\" (UID: \"87be3afe-a55b-48d2-a890-2788ea461965\") " Nov 03 00:33:47.357611 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.357582 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87be3afe-a55b-48d2-a890-2788ea461965-openshift-service-ca" pod "87be3afe-a55b-48d2-a890-2788ea461965" (UID: "87be3afe-a55b-48d2-a890-2788ea461965"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:47.357780 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.357751 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87be3afe-a55b-48d2-a890-2788ea461965-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:47.359365 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.359345 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87be3afe-a55b-48d2-a890-2788ea461965-kube-api-access-n82v2" pod "87be3afe-a55b-48d2-a890-2788ea461965" (UID: "87be3afe-a55b-48d2-a890-2788ea461965"). InnerVolumeSpecName "kube-api-access-n82v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:47.359462 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.359437 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-metrics-certs" pod "87be3afe-a55b-48d2-a890-2788ea461965" (UID: "87be3afe-a55b-48d2-a890-2788ea461965"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:33:47.359612 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.359599 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-maas-api-tls" pod "87be3afe-a55b-48d2-a890-2788ea461965" (UID: "87be3afe-a55b-48d2-a890-2788ea461965"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:33:47.458987 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.458924 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:47.458987 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.458947 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n82v2\" (UniqueName: \"kubernetes.io/projected/87be3afe-a55b-48d2-a890-2788ea461965-kube-api-access-n82v2\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:47.458987 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.458957 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87be3afe-a55b-48d2-a890-2788ea461965-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:47.572971 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.572939 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:47.727125 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.727051 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:47.822541 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.822509 6400 generic.go:356] "Generic (PLEG): container finished" podID="87be3afe-a55b-48d2-a890-2788ea461965" containerID="1af4a5d9a71ea35ebf47583531c2662ff4cf29f7db6b6003f081bd606192c1c0" exitCode=0 Nov 03 00:33:47.822678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.822556 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" Nov 03 00:33:47.822678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.822579 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" event={"ID":"87be3afe-a55b-48d2-a890-2788ea461965","Type":"ContainerDied","Data":"1af4a5d9a71ea35ebf47583531c2662ff4cf29f7db6b6003f081bd606192c1c0"} Nov 03 00:33:47.822678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.822649 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld" event={"ID":"87be3afe-a55b-48d2-a890-2788ea461965","Type":"ContainerDied","Data":"6ec5350cf64ed3d46d5f86ce4a231c2558526e5b2512370d68c1fffb3fa00fe9"} Nov 03 00:33:47.822678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.822673 6400 scope.go:122] "RemoveContainer" containerID="1af4a5d9a71ea35ebf47583531c2662ff4cf29f7db6b6003f081bd606192c1c0" Nov 03 00:33:47.832725 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.832710 6400 scope.go:122] "RemoveContainer" containerID="1af4a5d9a71ea35ebf47583531c2662ff4cf29f7db6b6003f081bd606192c1c0" Nov 03 00:33:47.833020 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:33:47.833001 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af4a5d9a71ea35ebf47583531c2662ff4cf29f7db6b6003f081bd606192c1c0\": container with ID starting with 1af4a5d9a71ea35ebf47583531c2662ff4cf29f7db6b6003f081bd606192c1c0 not found: ID does not exist" containerID="1af4a5d9a71ea35ebf47583531c2662ff4cf29f7db6b6003f081bd606192c1c0" Nov 03 00:33:47.833086 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:47.833030 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af4a5d9a71ea35ebf47583531c2662ff4cf29f7db6b6003f081bd606192c1c0"} err="failed to get container status \"1af4a5d9a71ea35ebf47583531c2662ff4cf29f7db6b6003f081bd606192c1c0\": rpc error: code = NotFound desc = could not find container \"1af4a5d9a71ea35ebf47583531c2662ff4cf29f7db6b6003f081bd606192c1c0\": container with ID starting with 1af4a5d9a71ea35ebf47583531c2662ff4cf29f7db6b6003f081bd606192c1c0 not found: ID does not exist" Nov 03 00:33:48.017583 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:48.017504 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld"] Nov 03 00:33:48.019632 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:48.019610 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-38f5a3-648cbc8bb4-rf6ld"] Nov 03 00:33:48.579307 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:48.579271 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:48.796549 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:48.796521 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:48.967034 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:48.966965 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:49.179830 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:49.179787 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87be3afe-a55b-48d2-a890-2788ea461965" path="/var/lib/kubelet/pods/87be3afe-a55b-48d2-a890-2788ea461965/volumes" Nov 03 00:33:50.068670 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:50.068632 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:50.590614 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:50.590581 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:54.553293 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:54.553258 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:55.778280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:55.778249 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:57.151944 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:33:57.151909 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:33:58.036355 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:58.036317 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:58.547380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:33:58.547348 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:01.123399 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:01.123358 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:01.613293 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:01.613245 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:05.125830 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:05.125780 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:34:05.990477 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:05.990435 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9"] Nov 03 00:34:05.991034 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:05.991015 6400 cpu_manager.go:414] "containerMap: removing container" podUID="87be3afe-a55b-48d2-a890-2788ea461965" containerName="maas-api" Nov 03 00:34:05.991138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:05.991035 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87be3afe-a55b-48d2-a890-2788ea461965" containerName="maas-api" Nov 03 00:34:05.991197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:05.991184 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="87be3afe-a55b-48d2-a890-2788ea461965" containerName="maas-api" Nov 03 00:34:05.997039 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:05.997017 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.099398 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.099366 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1a883c-serving-cert\"" Nov 03 00:34:06.101647 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.101625 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9"] Nov 03 00:34:06.106178 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.106146 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-metrics-certs\") pod \"maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.106313 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.106249 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dc7535-a699-4e56-bc44-b44bfa015d6f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.106479 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.106462 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.106544 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.106528 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk9fh\" (UniqueName: \"kubernetes.io/projected/98dc7535-a699-4e56-bc44-b44bfa015d6f-kube-api-access-zk9fh\") pod \"maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.204733 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.204703 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:06.208065 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.208044 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk9fh\" (UniqueName: \"kubernetes.io/projected/98dc7535-a699-4e56-bc44-b44bfa015d6f-kube-api-access-zk9fh\") pod \"maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.208148 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.208086 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-metrics-certs\") pod \"maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.208186 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.208152 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dc7535-a699-4e56-bc44-b44bfa015d6f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.208353 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.208337 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.208504 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:34:06.208482 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-serving-cert: secret "maas-api-e2e-ipp-a-1a883c-serving-cert" not found Nov 03 00:34:06.208581 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:34:06.208574 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-maas-api-tls podName:98dc7535-a699-4e56-bc44-b44bfa015d6f nodeName:}" failed. No retries permitted until 2026-11-03 00:34:06.708551978 +0000 UTC m=+1693.659492120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-maas-api-tls") pod "maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" (UID: "98dc7535-a699-4e56-bc44-b44bfa015d6f") : secret "maas-api-e2e-ipp-a-1a883c-serving-cert" not found Nov 03 00:34:06.208901 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.208881 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dc7535-a699-4e56-bc44-b44bfa015d6f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.210857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.210840 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-metrics-certs\") pod \"maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.297002 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.296970 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk9fh\" (UniqueName: \"kubernetes.io/projected/98dc7535-a699-4e56-bc44-b44bfa015d6f-kube-api-access-zk9fh\") pod \"maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.391224 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.391183 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:06.544735 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.544702 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:06.717573 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.717487 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.720069 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.720040 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:06.910946 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:06.910908 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:07.285470 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:07.285446 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9"] Nov 03 00:34:07.287127 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:34:07.287097 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98dc7535_a699_4e56_bc44_b44bfa015d6f.slice/crio-4948c531eb8a96e58ed9708612bc86569b9cb7d8c7deea97e36d4de144ef2123 WatchSource:0}: Error finding container 4948c531eb8a96e58ed9708612bc86569b9cb7d8c7deea97e36d4de144ef2123: Status 404 returned error can't find the container with id 4948c531eb8a96e58ed9708612bc86569b9cb7d8c7deea97e36d4de144ef2123 Nov 03 00:34:07.957913 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:07.957878 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" event={"ID":"98dc7535-a699-4e56-bc44-b44bfa015d6f","Type":"ContainerStarted","Data":"4948c531eb8a96e58ed9708612bc86569b9cb7d8c7deea97e36d4de144ef2123"} Nov 03 00:34:08.061973 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:08.061942 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:08.964959 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:08.964921 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" event={"ID":"98dc7535-a699-4e56-bc44-b44bfa015d6f","Type":"ContainerStarted","Data":"8ae3ad2ff7686d8f4048b7b7a6ce888085b1706f182c363b2946d394550b76dc"} Nov 03 00:34:08.965417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:08.964988 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:09.083083 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:09.083035 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" podStartSLOduration=3.211010561 podStartE2EDuration="4.083021986s" totalImagesPullingTime="872.011425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:09.081405539 +0000 UTC m=+1696.032345699" watchObservedRunningTime="2026-11-03 00:34:09.083021986 +0000 UTC m=+1696.033962143" Nov 03 00:34:11.523116 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:11.523076 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:14.977135 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:14.977108 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:34:15.105255 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:15.105218 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:15.662545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:15.662377 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:18.323224 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:18.323187 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:24.705030 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:24.704991 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:27.154672 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:34:27.154633 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:34:29.548292 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:29.548262 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:34.218355 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:34.218258 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:35.611440 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:35.611405 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:36.292335 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:36.292303 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:36.771504 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:36.771416 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:38.229174 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.229147 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2"] Nov 03 00:34:38.229557 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.229384 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" podUID="6daa92b6-9147-47f8-9cb2-c09b81d75359" containerName="maas-api" containerID="cri-o://caf0069b84e383e25c5a7a84b5b2f1e830bdfa412c8fcfba8a3cb7af89221390" gracePeriod=30 Nov 03 00:34:38.360184 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.360157 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:38.610943 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.610918 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:34:38.638752 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.638718 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-metrics-certs\") pod \"6daa92b6-9147-47f8-9cb2-c09b81d75359\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " Nov 03 00:34:38.639119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.639095 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-maas-api-tls\") pod \"6daa92b6-9147-47f8-9cb2-c09b81d75359\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " Nov 03 00:34:38.639267 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.639252 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6daa92b6-9147-47f8-9cb2-c09b81d75359-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6daa92b6-9147-47f8-9cb2-c09b81d75359-openshift-service-ca\") pod \"6daa92b6-9147-47f8-9cb2-c09b81d75359\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " Nov 03 00:34:38.639563 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.639547 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6daa92b6-9147-47f8-9cb2-c09b81d75359-kube-api-access-xrxc7\" (UniqueName: \"kubernetes.io/projected/6daa92b6-9147-47f8-9cb2-c09b81d75359-kube-api-access-xrxc7\") pod \"6daa92b6-9147-47f8-9cb2-c09b81d75359\" (UID: \"6daa92b6-9147-47f8-9cb2-c09b81d75359\") " Nov 03 00:34:38.651137 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.651084 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6daa92b6-9147-47f8-9cb2-c09b81d75359-openshift-service-ca" pod "6daa92b6-9147-47f8-9cb2-c09b81d75359" (UID: "6daa92b6-9147-47f8-9cb2-c09b81d75359"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:38.659222 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.659166 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6daa92b6-9147-47f8-9cb2-c09b81d75359-kube-api-access-xrxc7" pod "6daa92b6-9147-47f8-9cb2-c09b81d75359" (UID: "6daa92b6-9147-47f8-9cb2-c09b81d75359"). InnerVolumeSpecName "kube-api-access-xrxc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:38.686850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.675079 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-metrics-certs" pod "6daa92b6-9147-47f8-9cb2-c09b81d75359" (UID: "6daa92b6-9147-47f8-9cb2-c09b81d75359"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:34:38.686850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.675476 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-maas-api-tls" pod "6daa92b6-9147-47f8-9cb2-c09b81d75359" (UID: "6daa92b6-9147-47f8-9cb2-c09b81d75359"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:34:38.741613 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.741531 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrxc7\" (UniqueName: \"kubernetes.io/projected/6daa92b6-9147-47f8-9cb2-c09b81d75359-kube-api-access-xrxc7\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:38.741613 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.741560 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:38.741613 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.741571 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6daa92b6-9147-47f8-9cb2-c09b81d75359-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:38.741613 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:38.741579 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6daa92b6-9147-47f8-9cb2-c09b81d75359-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:39.152180 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.152149 6400 generic.go:356] "Generic (PLEG): container finished" podID="6daa92b6-9147-47f8-9cb2-c09b81d75359" containerID="caf0069b84e383e25c5a7a84b5b2f1e830bdfa412c8fcfba8a3cb7af89221390" exitCode=0 Nov 03 00:34:39.152332 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.152198 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" Nov 03 00:34:39.152332 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.152208 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" event={"ID":"6daa92b6-9147-47f8-9cb2-c09b81d75359","Type":"ContainerDied","Data":"caf0069b84e383e25c5a7a84b5b2f1e830bdfa412c8fcfba8a3cb7af89221390"} Nov 03 00:34:39.152332 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.152326 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2" event={"ID":"6daa92b6-9147-47f8-9cb2-c09b81d75359","Type":"ContainerDied","Data":"13f0af58d024197dbb7e0643e9599febb45c11e08b029936e40e9aeac61ad408"} Nov 03 00:34:39.152466 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.152346 6400 scope.go:122] "RemoveContainer" containerID="caf0069b84e383e25c5a7a84b5b2f1e830bdfa412c8fcfba8a3cb7af89221390" Nov 03 00:34:39.164077 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.164059 6400 scope.go:122] "RemoveContainer" containerID="caf0069b84e383e25c5a7a84b5b2f1e830bdfa412c8fcfba8a3cb7af89221390" Nov 03 00:34:39.164380 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:34:39.164361 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf0069b84e383e25c5a7a84b5b2f1e830bdfa412c8fcfba8a3cb7af89221390\": container with ID starting with caf0069b84e383e25c5a7a84b5b2f1e830bdfa412c8fcfba8a3cb7af89221390 not found: ID does not exist" containerID="caf0069b84e383e25c5a7a84b5b2f1e830bdfa412c8fcfba8a3cb7af89221390" Nov 03 00:34:39.164441 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.164389 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf0069b84e383e25c5a7a84b5b2f1e830bdfa412c8fcfba8a3cb7af89221390"} err="failed to get container status \"caf0069b84e383e25c5a7a84b5b2f1e830bdfa412c8fcfba8a3cb7af89221390\": rpc error: code = NotFound desc = could not find container \"caf0069b84e383e25c5a7a84b5b2f1e830bdfa412c8fcfba8a3cb7af89221390\": container with ID starting with caf0069b84e383e25c5a7a84b5b2f1e830bdfa412c8fcfba8a3cb7af89221390 not found: ID does not exist" Nov 03 00:34:39.325225 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.325194 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2"] Nov 03 00:34:39.373530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.373502 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb65f3a5-85fc5cbd4f-8vwk2"] Nov 03 00:34:39.895440 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.895404 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh"] Nov 03 00:34:39.895930 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.895916 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6daa92b6-9147-47f8-9cb2-c09b81d75359" containerName="maas-api" Nov 03 00:34:39.895995 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.895934 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6daa92b6-9147-47f8-9cb2-c09b81d75359" containerName="maas-api" Nov 03 00:34:39.896066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.896056 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6daa92b6-9147-47f8-9cb2-c09b81d75359" containerName="maas-api" Nov 03 00:34:39.900836 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.900785 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:39.955893 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.955866 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh"] Nov 03 00:34:39.958506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.958480 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-252tr\" (UniqueName: \"kubernetes.io/projected/deea825b-3387-4fae-bacf-88afa43dbae4-kube-api-access-252tr\") pod \"maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:39.958643 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.958536 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deea825b-3387-4fae-bacf-88afa43dbae4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:39.958643 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.958638 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:39.958773 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:39.958754 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-metrics-certs\") pod \"maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:40.003904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.003873 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-eb20df-serving-cert\"" Nov 03 00:34:40.060236 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.060210 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deea825b-3387-4fae-bacf-88afa43dbae4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:40.060379 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.060258 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:40.060379 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.060324 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-metrics-certs\") pod \"maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:40.060468 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.060396 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-252tr\" (UniqueName: \"kubernetes.io/projected/deea825b-3387-4fae-bacf-88afa43dbae4-kube-api-access-252tr\") pod \"maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:40.060468 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:34:40.060415 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-serving-cert: secret "maas-api-e2e-ipp-b-eb20df-serving-cert" not found Nov 03 00:34:40.060549 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:34:40.060493 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-maas-api-tls podName:deea825b-3387-4fae-bacf-88afa43dbae4 nodeName:}" failed. No retries permitted until 2026-11-03 00:34:40.560472089 +0000 UTC m=+1727.511412242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-maas-api-tls") pod "maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" (UID: "deea825b-3387-4fae-bacf-88afa43dbae4") : secret "maas-api-e2e-ipp-b-eb20df-serving-cert" not found Nov 03 00:34:40.060988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.060964 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deea825b-3387-4fae-bacf-88afa43dbae4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:40.062861 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.062845 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-metrics-certs\") pod \"maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:40.124841 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.118326 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-252tr\" (UniqueName: \"kubernetes.io/projected/deea825b-3387-4fae-bacf-88afa43dbae4-kube-api-access-252tr\") pod \"maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:40.137651 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.137622 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:40.377867 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.374385 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:40.570518 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.570360 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq"] Nov 03 00:34:40.570724 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.570654 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" podUID="079b7104-4e09-41de-b57d-79c1417f7336" containerName="istio-proxy" containerID="cri-o://95e8d6b468bb851467854c4f7928f3aba15386f06886fb98745ece680d7e8340" gracePeriod=30 Nov 03 00:34:40.575845 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.575805 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64"] Nov 03 00:34:40.592093 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.591616 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:40.593532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.592442 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" podUID="7c136a07-73a1-4f21-b543-e2ff13dc9bac" containerName="istio-proxy" containerID="cri-o://d4fcb308262ded728f78e73f1327481fcb3579f9572b6c388b36ff8982cdec2b" gracePeriod=30 Nov 03 00:34:40.595485 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.595394 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:40.595614 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.595594 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:40.764326 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.764235 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:40.814678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:40.814651 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:41.111958 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:41.111911 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:41.178710 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:41.178679 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6daa92b6-9147-47f8-9cb2-c09b81d75359" path="/var/lib/kubelet/pods/6daa92b6-9147-47f8-9cb2-c09b81d75359/volumes" Nov 03 00:34:41.191554 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:41.191535 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh"] Nov 03 00:34:41.193589 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:34:41.193566 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeea825b_3387_4fae_bacf_88afa43dbae4.slice/crio-8ad7f916db6487c79416a6cc65d8bab816e0239d5f1a21923611d0a66a3464bb WatchSource:0}: Error finding container 8ad7f916db6487c79416a6cc65d8bab816e0239d5f1a21923611d0a66a3464bb: Status 404 returned error can't find the container with id 8ad7f916db6487c79416a6cc65d8bab816e0239d5f1a21923611d0a66a3464bb Nov 03 00:34:41.283401 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:41.283366 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:41.462919 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:41.462797 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:41.639963 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:41.639932 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:41.812659 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:41.812632 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:42.174122 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:42.174095 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" event={"ID":"deea825b-3387-4fae-bacf-88afa43dbae4","Type":"ContainerStarted","Data":"8ad7f916db6487c79416a6cc65d8bab816e0239d5f1a21923611d0a66a3464bb"} Nov 03 00:34:42.710301 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:42.710268 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:42.934808 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:42.934775 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:43.181637 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:43.181597 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" event={"ID":"deea825b-3387-4fae-bacf-88afa43dbae4","Type":"ContainerStarted","Data":"b7f4daafe337012d180026039fe0b3ece57fc6246f724f2b6abce567ab3375ee"} Nov 03 00:34:43.181838 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:43.181731 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:43.309839 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:43.309768 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" podStartSLOduration=3.382468751 podStartE2EDuration="4.309753657s" totalImagesPullingTime="927.284906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:43.308517637 +0000 UTC m=+1730.259457982" watchObservedRunningTime="2026-11-03 00:34:43.309753657 +0000 UTC m=+1730.260693815" Nov 03 00:34:44.407564 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:44.407531 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:44.617403 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:44.617359 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:44.831582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:44.827623 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:44.996702 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:44.996667 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:45.727603 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.727568 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:45.906094 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.906071 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:34:45.948845 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.948779 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:34:45.983586 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.983556 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-token\" (UniqueName: \"kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-token\") pod \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " Nov 03 00:34:45.983758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.983624 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-data\") pod \"079b7104-4e09-41de-b57d-79c1417f7336\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " Nov 03 00:34:45.983758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.983643 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-podinfo\") pod \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " Nov 03 00:34:45.983758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.983662 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-istio-token\" (UniqueName: \"kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-istio-token\") pod \"079b7104-4e09-41de-b57d-79c1417f7336\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " Nov 03 00:34:45.983758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.983678 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-data\") pod \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " Nov 03 00:34:45.983758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.983697 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-kube-api-access-4fxdn\" (UniqueName: \"kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-kube-api-access-4fxdn\") pod \"079b7104-4e09-41de-b57d-79c1417f7336\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " Nov 03 00:34:45.983758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.983730 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-certs\") pod \"079b7104-4e09-41de-b57d-79c1417f7336\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " Nov 03 00:34:45.984087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.983809 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/079b7104-4e09-41de-b57d-79c1417f7336-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079b7104-4e09-41de-b57d-79c1417f7336-istio-podinfo\") pod \"079b7104-4e09-41de-b57d-79c1417f7336\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " Nov 03 00:34:45.984087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.983850 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-envoy\") pod \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " Nov 03 00:34:45.984087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.983882 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istiod-ca-cert\") pod \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " Nov 03 00:34:45.984087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.983941 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-socket\") pod \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " Nov 03 00:34:45.984087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.983972 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-kube-api-access-6sql4\" (UniqueName: \"kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-kube-api-access-6sql4\") pod \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " Nov 03 00:34:45.984087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984009 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-certs\") pod \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " Nov 03 00:34:45.984087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984022 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-data" pod "7c136a07-73a1-4f21-b543-e2ff13dc9bac" (UID: "7c136a07-73a1-4f21-b543-e2ff13dc9bac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.984087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984035 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-data" pod "079b7104-4e09-41de-b57d-79c1417f7336" (UID: "079b7104-4e09-41de-b57d-79c1417f7336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.984087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984055 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-credential-socket\") pod \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\" (UID: \"7c136a07-73a1-4f21-b543-e2ff13dc9bac\") " Nov 03 00:34:45.984532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984116 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/079b7104-4e09-41de-b57d-79c1417f7336-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079b7104-4e09-41de-b57d-79c1417f7336-istiod-ca-cert\") pod \"079b7104-4e09-41de-b57d-79c1417f7336\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " Nov 03 00:34:45.984532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984149 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-envoy\") pod \"079b7104-4e09-41de-b57d-79c1417f7336\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " Nov 03 00:34:45.984532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984193 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-credential-socket\") pod \"079b7104-4e09-41de-b57d-79c1417f7336\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " Nov 03 00:34:45.984532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984222 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-socket\") pod \"079b7104-4e09-41de-b57d-79c1417f7336\" (UID: \"079b7104-4e09-41de-b57d-79c1417f7336\") " Nov 03 00:34:45.984532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984223 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-socket" pod "7c136a07-73a1-4f21-b543-e2ff13dc9bac" (UID: "7c136a07-73a1-4f21-b543-e2ff13dc9bac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.984532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984236 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-certs" pod "079b7104-4e09-41de-b57d-79c1417f7336" (UID: "079b7104-4e09-41de-b57d-79c1417f7336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.984532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984416 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-credential-socket" pod "7c136a07-73a1-4f21-b543-e2ff13dc9bac" (UID: "7c136a07-73a1-4f21-b543-e2ff13dc9bac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.984532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984518 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istiod-ca-cert" pod "7c136a07-73a1-4f21-b543-e2ff13dc9bac" (UID: "7c136a07-73a1-4f21-b543-e2ff13dc9bac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:45.984975 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984689 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-socket" pod "079b7104-4e09-41de-b57d-79c1417f7336" (UID: "079b7104-4e09-41de-b57d-79c1417f7336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.984975 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984911 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.984975 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984933 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.984975 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984948 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.984975 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984963 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.985414 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984977 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.985414 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.984993 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.985414 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.985006 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.985414 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.985073 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-certs" pod "7c136a07-73a1-4f21-b543-e2ff13dc9bac" (UID: "7c136a07-73a1-4f21-b543-e2ff13dc9bac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.985414 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.985090 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-credential-socket" pod "079b7104-4e09-41de-b57d-79c1417f7336" (UID: "079b7104-4e09-41de-b57d-79c1417f7336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.985414 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.985145 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079b7104-4e09-41de-b57d-79c1417f7336-istiod-ca-cert" pod "079b7104-4e09-41de-b57d-79c1417f7336" (UID: "079b7104-4e09-41de-b57d-79c1417f7336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:45.986736 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.986711 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-podinfo" pod "7c136a07-73a1-4f21-b543-e2ff13dc9bac" (UID: "7c136a07-73a1-4f21-b543-e2ff13dc9bac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:34:45.987089 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.987021 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-istio-token" pod "079b7104-4e09-41de-b57d-79c1417f7336" (UID: "079b7104-4e09-41de-b57d-79c1417f7336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:45.987171 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.987148 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/079b7104-4e09-41de-b57d-79c1417f7336-istio-podinfo" pod "079b7104-4e09-41de-b57d-79c1417f7336" (UID: "079b7104-4e09-41de-b57d-79c1417f7336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:34:45.987224 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.987186 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-kube-api-access-6sql4" pod "7c136a07-73a1-4f21-b543-e2ff13dc9bac" (UID: "7c136a07-73a1-4f21-b543-e2ff13dc9bac"). InnerVolumeSpecName "kube-api-access-6sql4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:45.987279 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.987260 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-token" pod "7c136a07-73a1-4f21-b543-e2ff13dc9bac" (UID: "7c136a07-73a1-4f21-b543-e2ff13dc9bac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:45.987315 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.987289 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-kube-api-access-4fxdn" pod "079b7104-4e09-41de-b57d-79c1417f7336" (UID: "079b7104-4e09-41de-b57d-79c1417f7336"). InnerVolumeSpecName "kube-api-access-4fxdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:45.987587 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.987564 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-envoy" pod "7c136a07-73a1-4f21-b543-e2ff13dc9bac" (UID: "7c136a07-73a1-4f21-b543-e2ff13dc9bac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.988354 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:45.988335 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-envoy" pod "079b7104-4e09-41de-b57d-79c1417f7336" (UID: "079b7104-4e09-41de-b57d-79c1417f7336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:46.086154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.086121 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.086154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.086151 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.086154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.086161 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.086154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.086170 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fxdn\" (UniqueName: \"kubernetes.io/projected/079b7104-4e09-41de-b57d-79c1417f7336-kube-api-access-4fxdn\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.086427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.086181 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079b7104-4e09-41de-b57d-79c1417f7336-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.086427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.086190 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.086427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.086198 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sql4\" (UniqueName: \"kubernetes.io/projected/7c136a07-73a1-4f21-b543-e2ff13dc9bac-kube-api-access-6sql4\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.086427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.086206 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c136a07-73a1-4f21-b543-e2ff13dc9bac-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.086427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.086214 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079b7104-4e09-41de-b57d-79c1417f7336-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.086427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.086223 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.086427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.086231 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079b7104-4e09-41de-b57d-79c1417f7336-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.265265 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.265179 6400 generic.go:356] "Generic (PLEG): container finished" podID="7c136a07-73a1-4f21-b543-e2ff13dc9bac" containerID="d4fcb308262ded728f78e73f1327481fcb3579f9572b6c388b36ff8982cdec2b" exitCode=0 Nov 03 00:34:46.265265 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.265226 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" Nov 03 00:34:46.265467 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.265275 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" event={"ID":"7c136a07-73a1-4f21-b543-e2ff13dc9bac","Type":"ContainerDied","Data":"d4fcb308262ded728f78e73f1327481fcb3579f9572b6c388b36ff8982cdec2b"} Nov 03 00:34:46.265467 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.265332 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64" event={"ID":"7c136a07-73a1-4f21-b543-e2ff13dc9bac","Type":"ContainerDied","Data":"aea08c9273718c4ba3c799c5705bbc34e797694317bc2ec1619c109105eb7f13"} Nov 03 00:34:46.265467 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.265364 6400 scope.go:122] "RemoveContainer" containerID="d4fcb308262ded728f78e73f1327481fcb3579f9572b6c388b36ff8982cdec2b" Nov 03 00:34:46.266976 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.266943 6400 generic.go:356] "Generic (PLEG): container finished" podID="079b7104-4e09-41de-b57d-79c1417f7336" containerID="95e8d6b468bb851467854c4f7928f3aba15386f06886fb98745ece680d7e8340" exitCode=0 Nov 03 00:34:46.267383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.266980 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" Nov 03 00:34:46.267383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.266995 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" event={"ID":"079b7104-4e09-41de-b57d-79c1417f7336","Type":"ContainerDied","Data":"95e8d6b468bb851467854c4f7928f3aba15386f06886fb98745ece680d7e8340"} Nov 03 00:34:46.267383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.267029 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq" event={"ID":"079b7104-4e09-41de-b57d-79c1417f7336","Type":"ContainerDied","Data":"1230ae517f4280026167ea44866e35a08535d223a9cf6275f36571a828d9105e"} Nov 03 00:34:46.276392 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.276369 6400 scope.go:122] "RemoveContainer" containerID="d4fcb308262ded728f78e73f1327481fcb3579f9572b6c388b36ff8982cdec2b" Nov 03 00:34:46.276777 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:34:46.276737 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fcb308262ded728f78e73f1327481fcb3579f9572b6c388b36ff8982cdec2b\": container with ID starting with d4fcb308262ded728f78e73f1327481fcb3579f9572b6c388b36ff8982cdec2b not found: ID does not exist" containerID="d4fcb308262ded728f78e73f1327481fcb3579f9572b6c388b36ff8982cdec2b" Nov 03 00:34:46.276893 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.276809 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fcb308262ded728f78e73f1327481fcb3579f9572b6c388b36ff8982cdec2b"} err="failed to get container status \"d4fcb308262ded728f78e73f1327481fcb3579f9572b6c388b36ff8982cdec2b\": rpc error: code = NotFound desc = could not find container \"d4fcb308262ded728f78e73f1327481fcb3579f9572b6c388b36ff8982cdec2b\": container with ID starting with d4fcb308262ded728f78e73f1327481fcb3579f9572b6c388b36ff8982cdec2b not found: ID does not exist" Nov 03 00:34:46.276893 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.276871 6400 scope.go:122] "RemoveContainer" containerID="95e8d6b468bb851467854c4f7928f3aba15386f06886fb98745ece680d7e8340" Nov 03 00:34:46.285916 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.285881 6400 scope.go:122] "RemoveContainer" containerID="95e8d6b468bb851467854c4f7928f3aba15386f06886fb98745ece680d7e8340" Nov 03 00:34:46.286207 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:34:46.286181 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e8d6b468bb851467854c4f7928f3aba15386f06886fb98745ece680d7e8340\": container with ID starting with 95e8d6b468bb851467854c4f7928f3aba15386f06886fb98745ece680d7e8340 not found: ID does not exist" containerID="95e8d6b468bb851467854c4f7928f3aba15386f06886fb98745ece680d7e8340" Nov 03 00:34:46.286251 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.286216 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e8d6b468bb851467854c4f7928f3aba15386f06886fb98745ece680d7e8340"} err="failed to get container status \"95e8d6b468bb851467854c4f7928f3aba15386f06886fb98745ece680d7e8340\": rpc error: code = NotFound desc = could not find container \"95e8d6b468bb851467854c4f7928f3aba15386f06886fb98745ece680d7e8340\": container with ID starting with 95e8d6b468bb851467854c4f7928f3aba15386f06886fb98745ece680d7e8340 not found: ID does not exist" Nov 03 00:34:46.432268 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.432235 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq"] Nov 03 00:34:46.436289 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.436264 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fktgq"] Nov 03 00:34:46.544744 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.544715 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64"] Nov 03 00:34:46.553354 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:46.553323 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-fnb64"] Nov 03 00:34:47.179950 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:47.179917 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="079b7104-4e09-41de-b57d-79c1417f7336" path="/var/lib/kubelet/pods/079b7104-4e09-41de-b57d-79c1417f7336/volumes" Nov 03 00:34:47.180377 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:47.180364 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c136a07-73a1-4f21-b543-e2ff13dc9bac" path="/var/lib/kubelet/pods/7c136a07-73a1-4f21-b543-e2ff13dc9bac/volumes" Nov 03 00:34:49.194363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:49.194335 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:34:51.108933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:51.108898 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:51.684724 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:51.684690 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:34:55.600415 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:55.600381 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:55.822375 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:55.811660 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg"] Nov 03 00:34:55.822375 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:55.811936 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" podUID="4e3cd93c-0280-450f-9445-eab9644817db" containerName="istio-proxy" containerID="cri-o://95d7194da63e0b0e770ac9fb74ec484be9d9996823e4a5743defd47cc8c6daff" gracePeriod=30 Nov 03 00:34:56.210050 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:56.209756 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:56.407408 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:56.407363 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:57.152928 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:34:57.152891 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:34:57.544964 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:57.544929 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:58.483484 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:58.483442 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl"] Nov 03 00:34:58.483948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:58.483710 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" podUID="4e7ab390-893f-4d69-982a-464bf5483fa5" containerName="istio-proxy" containerID="cri-o://be9998f569960e28d32cb0ae4093f845f3fa61e43e931aeb2b5329756307e93f" gracePeriod=30 Nov 03 00:34:59.677872 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:59.677804 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:34:59.678255 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:34:59.677894 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" podUID="4e3cd93c-0280-450f-9445-eab9644817db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:35:00.390161 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:00.390127 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:01.132034 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.132006 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:35:01.243508 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.243445 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-envoy\") pod \"4e3cd93c-0280-450f-9445-eab9644817db\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " Nov 03 00:35:01.243508 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.243498 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e3cd93c-0280-450f-9445-eab9644817db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3cd93c-0280-450f-9445-eab9644817db-istiod-ca-cert\") pod \"4e3cd93c-0280-450f-9445-eab9644817db\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " Nov 03 00:35:01.243695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.243543 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-credential-socket\") pod \"4e3cd93c-0280-450f-9445-eab9644817db\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " Nov 03 00:35:01.243695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.243570 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e3cd93c-0280-450f-9445-eab9644817db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3cd93c-0280-450f-9445-eab9644817db-istio-podinfo\") pod \"4e3cd93c-0280-450f-9445-eab9644817db\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " Nov 03 00:35:01.243695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.243631 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-certs\") pod \"4e3cd93c-0280-450f-9445-eab9644817db\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " Nov 03 00:35:01.243695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.243689 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-socket\") pod \"4e3cd93c-0280-450f-9445-eab9644817db\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " Nov 03 00:35:01.244005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.243726 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-istio-token\" (UniqueName: \"kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-istio-token\") pod \"4e3cd93c-0280-450f-9445-eab9644817db\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " Nov 03 00:35:01.244005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.243854 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-credential-socket" pod "4e3cd93c-0280-450f-9445-eab9644817db" (UID: "4e3cd93c-0280-450f-9445-eab9644817db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:01.244119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.244014 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-certs" pod "4e3cd93c-0280-450f-9445-eab9644817db" (UID: "4e3cd93c-0280-450f-9445-eab9644817db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:01.244119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.244060 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3cd93c-0280-450f-9445-eab9644817db-istiod-ca-cert" pod "4e3cd93c-0280-450f-9445-eab9644817db" (UID: "4e3cd93c-0280-450f-9445-eab9644817db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:01.244226 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.243757 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-kube-api-access-t8kz7\" (UniqueName: \"kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-kube-api-access-t8kz7\") pod \"4e3cd93c-0280-450f-9445-eab9644817db\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " Nov 03 00:35:01.244286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.244248 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-data\") pod \"4e3cd93c-0280-450f-9445-eab9644817db\" (UID: \"4e3cd93c-0280-450f-9445-eab9644817db\") " Nov 03 00:35:01.244361 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.244331 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-socket" pod "4e3cd93c-0280-450f-9445-eab9644817db" (UID: "4e3cd93c-0280-450f-9445-eab9644817db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:01.244854 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.244630 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-data" pod "4e3cd93c-0280-450f-9445-eab9644817db" (UID: "4e3cd93c-0280-450f-9445-eab9644817db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:01.245205 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.245141 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:01.245205 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.245165 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3cd93c-0280-450f-9445-eab9644817db-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:01.245205 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.245181 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:01.245205 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.245196 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:01.245459 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.245210 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:01.246599 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.246562 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-kube-api-access-t8kz7" pod "4e3cd93c-0280-450f-9445-eab9644817db" (UID: "4e3cd93c-0280-450f-9445-eab9644817db"). InnerVolumeSpecName "kube-api-access-t8kz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:01.246839 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.246795 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-envoy" pod "4e3cd93c-0280-450f-9445-eab9644817db" (UID: "4e3cd93c-0280-450f-9445-eab9644817db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:01.246963 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.246943 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-istio-token" pod "4e3cd93c-0280-450f-9445-eab9644817db" (UID: "4e3cd93c-0280-450f-9445-eab9644817db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:01.247242 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.247222 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e3cd93c-0280-450f-9445-eab9644817db-istio-podinfo" pod "4e3cd93c-0280-450f-9445-eab9644817db" (UID: "4e3cd93c-0280-450f-9445-eab9644817db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:35:01.346241 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.346208 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:01.346366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.346248 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8kz7\" (UniqueName: \"kubernetes.io/projected/4e3cd93c-0280-450f-9445-eab9644817db-kube-api-access-t8kz7\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:01.346366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.346267 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd93c-0280-450f-9445-eab9644817db-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:01.346366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.346281 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3cd93c-0280-450f-9445-eab9644817db-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:01.399962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.399928 6400 generic.go:356] "Generic (PLEG): container finished" podID="4e3cd93c-0280-450f-9445-eab9644817db" containerID="95d7194da63e0b0e770ac9fb74ec484be9d9996823e4a5743defd47cc8c6daff" exitCode=0 Nov 03 00:35:01.400110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.399977 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" Nov 03 00:35:01.400110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.400015 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" event={"ID":"4e3cd93c-0280-450f-9445-eab9644817db","Type":"ContainerDied","Data":"95d7194da63e0b0e770ac9fb74ec484be9d9996823e4a5743defd47cc8c6daff"} Nov 03 00:35:01.400110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.400073 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg" event={"ID":"4e3cd93c-0280-450f-9445-eab9644817db","Type":"ContainerDied","Data":"7a0a06920eddeb083e83d07f9fb3958572082f10d3d01767e7ba5588a98d5fe8"} Nov 03 00:35:01.400110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.400096 6400 scope.go:122] "RemoveContainer" containerID="95d7194da63e0b0e770ac9fb74ec484be9d9996823e4a5743defd47cc8c6daff" Nov 03 00:35:01.415104 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.415083 6400 scope.go:122] "RemoveContainer" containerID="95d7194da63e0b0e770ac9fb74ec484be9d9996823e4a5743defd47cc8c6daff" Nov 03 00:35:01.415418 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:35:01.415394 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d7194da63e0b0e770ac9fb74ec484be9d9996823e4a5743defd47cc8c6daff\": container with ID starting with 95d7194da63e0b0e770ac9fb74ec484be9d9996823e4a5743defd47cc8c6daff not found: ID does not exist" containerID="95d7194da63e0b0e770ac9fb74ec484be9d9996823e4a5743defd47cc8c6daff" Nov 03 00:35:01.415502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.415427 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d7194da63e0b0e770ac9fb74ec484be9d9996823e4a5743defd47cc8c6daff"} err="failed to get container status \"95d7194da63e0b0e770ac9fb74ec484be9d9996823e4a5743defd47cc8c6daff\": rpc error: code = NotFound desc = could not find container \"95d7194da63e0b0e770ac9fb74ec484be9d9996823e4a5743defd47cc8c6daff\": container with ID starting with 95d7194da63e0b0e770ac9fb74ec484be9d9996823e4a5743defd47cc8c6daff not found: ID does not exist" Nov 03 00:35:01.566222 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.566193 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg"] Nov 03 00:35:01.570428 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.570404 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-jp4vg"] Nov 03 00:35:01.698535 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.698504 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:01.907361 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:01.907333 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:02.186547 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:02.186476 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:35:02.186547 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:02.186529 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" podUID="4e7ab390-893f-4d69-982a-464bf5483fa5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:35:02.315440 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:02.315404 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:02.407508 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:02.407477 6400 generic.go:356] "Generic (PLEG): container finished" podID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerID="b94b032c32511accf166739d2f73eaa6d6b8ab2e7489de9e4364df3923116859" exitCode=0 Nov 03 00:35:02.407665 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:02.407543 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" event={"ID":"1e679802-f6b3-49e5-9c52-f523957ce3ed","Type":"ContainerDied","Data":"b94b032c32511accf166739d2f73eaa6d6b8ab2e7489de9e4364df3923116859"} Nov 03 00:35:02.407980 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:02.407964 6400 scope.go:122] "RemoveContainer" containerID="b94b032c32511accf166739d2f73eaa6d6b8ab2e7489de9e4364df3923116859" Nov 03 00:35:02.409401 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:02.409375 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:35:02.409470 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:02.409443 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:35:02.409512 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:02.409473 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:35:03.178457 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.178427 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e3cd93c-0280-450f-9445-eab9644817db" path="/var/lib/kubelet/pods/4e3cd93c-0280-450f-9445-eab9644817db/volumes" Nov 03 00:35:03.219266 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.219238 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:03.414704 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.414671 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" event={"ID":"1e679802-f6b3-49e5-9c52-f523957ce3ed","Type":"ContainerStarted","Data":"935d03468dec41fa8aca4ff048ce65e47a6618662c76e1f8daf49d22e9b10917"} Nov 03 00:35:03.727886 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.727857 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:03.773498 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.773470 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:35:03.877903 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.877874 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-data\") pod \"4e7ab390-893f-4d69-982a-464bf5483fa5\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " Nov 03 00:35:03.878104 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.877914 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-token\" (UniqueName: \"kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-token\") pod \"4e7ab390-893f-4d69-982a-464bf5483fa5\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " Nov 03 00:35:03.878185 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.878170 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-podinfo\") pod \"4e7ab390-893f-4d69-982a-464bf5483fa5\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " Nov 03 00:35:03.878260 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.878243 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7ab390-893f-4d69-982a-464bf5483fa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7ab390-893f-4d69-982a-464bf5483fa5-istiod-ca-cert\") pod \"4e7ab390-893f-4d69-982a-464bf5483fa5\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " Nov 03 00:35:03.878318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.878275 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-credential-socket\") pod \"4e7ab390-893f-4d69-982a-464bf5483fa5\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " Nov 03 00:35:03.878318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.878312 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-envoy\") pod \"4e7ab390-893f-4d69-982a-464bf5483fa5\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " Nov 03 00:35:03.878426 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.878347 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-socket\") pod \"4e7ab390-893f-4d69-982a-464bf5483fa5\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " Nov 03 00:35:03.878426 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.878393 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-certs\") pod \"4e7ab390-893f-4d69-982a-464bf5483fa5\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " Nov 03 00:35:03.878527 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.878456 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-kube-api-access-ggrh5\" (UniqueName: \"kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-kube-api-access-ggrh5\") pod \"4e7ab390-893f-4d69-982a-464bf5483fa5\" (UID: \"4e7ab390-893f-4d69-982a-464bf5483fa5\") " Nov 03 00:35:03.878527 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.878478 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-credential-socket" pod "4e7ab390-893f-4d69-982a-464bf5483fa5" (UID: "4e7ab390-893f-4d69-982a-464bf5483fa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:03.878527 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.878497 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-data" pod "4e7ab390-893f-4d69-982a-464bf5483fa5" (UID: "4e7ab390-893f-4d69-982a-464bf5483fa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:03.878687 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.878555 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7ab390-893f-4d69-982a-464bf5483fa5-istiod-ca-cert" pod "4e7ab390-893f-4d69-982a-464bf5483fa5" (UID: "4e7ab390-893f-4d69-982a-464bf5483fa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:03.878765 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.878742 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-socket" pod "4e7ab390-893f-4d69-982a-464bf5483fa5" (UID: "4e7ab390-893f-4d69-982a-464bf5483fa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:03.878765 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.878759 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-certs" pod "4e7ab390-893f-4d69-982a-464bf5483fa5" (UID: "4e7ab390-893f-4d69-982a-464bf5483fa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:03.879014 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.878993 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7ab390-893f-4d69-982a-464bf5483fa5-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.879090 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.879021 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.879090 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.879037 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.879090 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.879049 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.879090 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.879064 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.881134 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.881108 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-podinfo" pod "4e7ab390-893f-4d69-982a-464bf5483fa5" (UID: "4e7ab390-893f-4d69-982a-464bf5483fa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:35:03.881412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.881390 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-envoy" pod "4e7ab390-893f-4d69-982a-464bf5483fa5" (UID: "4e7ab390-893f-4d69-982a-464bf5483fa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:03.881528 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.881464 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-token" pod "4e7ab390-893f-4d69-982a-464bf5483fa5" (UID: "4e7ab390-893f-4d69-982a-464bf5483fa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:03.881597 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.881522 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-kube-api-access-ggrh5" pod "4e7ab390-893f-4d69-982a-464bf5483fa5" (UID: "4e7ab390-893f-4d69-982a-464bf5483fa5"). InnerVolumeSpecName "kube-api-access-ggrh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:03.961500 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.961470 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:35:03.961834 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.961790 6400 patch_prober.go:29] interesting pod/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:35:03.961967 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.961871 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 00:35:03.980282 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.980151 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.980282 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.980187 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.980282 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.980202 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggrh5\" (UniqueName: \"kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-kube-api-access-ggrh5\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:03.980282 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:03.980215 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7ab390-893f-4d69-982a-464bf5483fa5-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:04.421306 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:04.421274 6400 generic.go:356] "Generic (PLEG): container finished" podID="4e7ab390-893f-4d69-982a-464bf5483fa5" containerID="be9998f569960e28d32cb0ae4093f845f3fa61e43e931aeb2b5329756307e93f" exitCode=0 Nov 03 00:35:04.421772 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:04.421354 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" Nov 03 00:35:04.421772 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:04.421367 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" event={"ID":"4e7ab390-893f-4d69-982a-464bf5483fa5","Type":"ContainerDied","Data":"be9998f569960e28d32cb0ae4093f845f3fa61e43e931aeb2b5329756307e93f"} Nov 03 00:35:04.421772 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:04.421462 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl" event={"ID":"4e7ab390-893f-4d69-982a-464bf5483fa5","Type":"ContainerDied","Data":"8d683e0c9a70f03ed07e1486bdc1c2b932abfd118ee1d7fc0787ba3f06ffe6cc"} Nov 03 00:35:04.421772 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:04.421491 6400 scope.go:122] "RemoveContainer" containerID="be9998f569960e28d32cb0ae4093f845f3fa61e43e931aeb2b5329756307e93f" Nov 03 00:35:04.432461 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:04.432440 6400 scope.go:122] "RemoveContainer" containerID="be9998f569960e28d32cb0ae4093f845f3fa61e43e931aeb2b5329756307e93f" Nov 03 00:35:04.432780 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:35:04.432759 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9998f569960e28d32cb0ae4093f845f3fa61e43e931aeb2b5329756307e93f\": container with ID starting with be9998f569960e28d32cb0ae4093f845f3fa61e43e931aeb2b5329756307e93f not found: ID does not exist" containerID="be9998f569960e28d32cb0ae4093f845f3fa61e43e931aeb2b5329756307e93f" Nov 03 00:35:04.432882 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:04.432788 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9998f569960e28d32cb0ae4093f845f3fa61e43e931aeb2b5329756307e93f"} err="failed to get container status \"be9998f569960e28d32cb0ae4093f845f3fa61e43e931aeb2b5329756307e93f\": rpc error: code = NotFound desc = could not find container \"be9998f569960e28d32cb0ae4093f845f3fa61e43e931aeb2b5329756307e93f\": container with ID starting with be9998f569960e28d32cb0ae4093f845f3fa61e43e931aeb2b5329756307e93f not found: ID does not exist" Nov 03 00:35:04.601784 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:04.601734 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl"] Nov 03 00:35:04.649906 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:04.649874 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-p5rhl"] Nov 03 00:35:04.961576 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:04.961541 6400 patch_prober.go:29] interesting pod/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:35:04.961879 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:04.961593 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 00:35:05.200846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.200266 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7ab390-893f-4d69-982a-464bf5483fa5" path="/var/lib/kubelet/pods/4e7ab390-893f-4d69-982a-464bf5483fa5/volumes" Nov 03 00:35:05.258708 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.258615 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:05.803847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.803480 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd"] Nov 03 00:35:05.806341 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.806300 6400 cpu_manager.go:414] "containerMap: removing container" podUID="079b7104-4e09-41de-b57d-79c1417f7336" containerName="istio-proxy" Nov 03 00:35:05.806341 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.806333 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079b7104-4e09-41de-b57d-79c1417f7336" containerName="istio-proxy" Nov 03 00:35:05.806545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.806365 6400 cpu_manager.go:414] "containerMap: removing container" podUID="7c136a07-73a1-4f21-b543-e2ff13dc9bac" containerName="istio-proxy" Nov 03 00:35:05.806545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.806373 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c136a07-73a1-4f21-b543-e2ff13dc9bac" containerName="istio-proxy" Nov 03 00:35:05.806545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.806391 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4e7ab390-893f-4d69-982a-464bf5483fa5" containerName="istio-proxy" Nov 03 00:35:05.806545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.806398 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7ab390-893f-4d69-982a-464bf5483fa5" containerName="istio-proxy" Nov 03 00:35:05.806545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.806416 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4e3cd93c-0280-450f-9445-eab9644817db" containerName="istio-proxy" Nov 03 00:35:05.806545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.806424 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3cd93c-0280-450f-9445-eab9644817db" containerName="istio-proxy" Nov 03 00:35:05.806898 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.806556 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7ab390-893f-4d69-982a-464bf5483fa5" containerName="istio-proxy" Nov 03 00:35:05.806898 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.806568 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c136a07-73a1-4f21-b543-e2ff13dc9bac" containerName="istio-proxy" Nov 03 00:35:05.806898 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.806577 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3cd93c-0280-450f-9445-eab9644817db" containerName="istio-proxy" Nov 03 00:35:05.806898 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.806590 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="079b7104-4e09-41de-b57d-79c1417f7336" containerName="istio-proxy" Nov 03 00:35:05.811424 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.811406 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:05.866853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.865885 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-cb486a-openshift-default-dockercfg-fv5ql\"" Nov 03 00:35:05.925899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.923814 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-credential-socket\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:05.925899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.923901 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-envoy\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:05.925899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.923936 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-podinfo\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:05.925899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.923987 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-certs\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:05.937851 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.924095 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istiod-ca-cert\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:05.938043 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.937883 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjpp6\" (UniqueName: \"kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-kube-api-access-cjpp6\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:05.938043 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.937927 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-data\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:05.938043 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.937957 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-socket\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:05.938043 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:05.937988 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-token\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.021845 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.021420 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd"] Nov 03 00:35:06.058904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.058189 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istiod-ca-cert\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.065993 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.065954 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjpp6\" (UniqueName: \"kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-kube-api-access-cjpp6\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.066168 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.066031 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-data\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.066168 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.066068 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-socket\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.066168 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.066120 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-token\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.066350 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.066197 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-credential-socket\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.066350 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.066273 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-envoy\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.066350 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.066326 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-podinfo\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.066607 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.066585 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-certs\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.068018 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.067987 6400 patch_prober.go:29] interesting pod/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:35:06.068149 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.068047 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 00:35:06.068655 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.068627 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istiod-ca-cert\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.068930 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.068911 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:06.069438 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.069414 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-certs\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.069784 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.069756 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-data\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.070018 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.069998 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-socket\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.070226 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.070208 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-credential-socket\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.074349 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.074106 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-envoy\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.074349 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.074322 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-podinfo\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.128536 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.128472 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-token\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.129605 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.129575 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjpp6\" (UniqueName: \"kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-kube-api-access-cjpp6\") pod \"e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.299507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.299448 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:06.429093 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.428997 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:06.894801 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.882299 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:06.894801 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.892840 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd"] Nov 03 00:35:06.922079 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.917795 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:35:06.922079 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.917907 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:35:06.922079 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.917971 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:35:06.961308 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.961275 6400 patch_prober.go:29] interesting pod/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:35:06.961459 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:06.961335 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 00:35:07.442301 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:07.442267 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" event={"ID":"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53","Type":"ContainerStarted","Data":"aa42b5d0eb09dc1a0e6155b9ff15a59b978afdd8297081b8665cd0ee3cfce650"} Nov 03 00:35:07.442484 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:07.442316 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" event={"ID":"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53","Type":"ContainerStarted","Data":"3124b72012d183b8f8b07a5931a06ebbfaf5e6d3a4965157e51da6e8451fd4cc"} Nov 03 00:35:07.561960 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:07.561889 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" podStartSLOduration=2.561871184 podStartE2EDuration="2.561871184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:07.560657302 +0000 UTC m=+1754.511597459" watchObservedRunningTime="2026-11-03 00:35:07.561871184 +0000 UTC m=+1754.512811342" Nov 03 00:35:07.893845 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:07.892564 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:07.961237 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:07.961206 6400 patch_prober.go:29] interesting pod/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:35:07.961650 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:07.961259 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 00:35:08.328349 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:08.328313 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:08.429460 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:08.429427 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:08.435519 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:08.435494 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:08.457005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:08.456953 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:08.458086 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:08.458063 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:35:08.961539 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:08.961504 6400 patch_prober.go:29] interesting pod/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:35:08.961963 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:08.961558 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 00:35:09.966661 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:09.966633 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:35:09.967086 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:09.967034 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:35:09.967982 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:09.967963 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:35:10.397351 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:10.397316 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:10.570900 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:10.570865 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:10.745922 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:10.745807 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:12.330960 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:12.330930 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:12.551708 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:12.551677 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:12.722010 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:12.721922 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:14.927974 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:14.927942 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:16.095949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:16.095916 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:16.316060 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:16.316024 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:21.917658 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:21.917620 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:26.810660 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:26.810628 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:27.151546 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:35:27.151464 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:35:30.012375 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:30.012339 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:35:31.489422 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:31.489386 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:35.969003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:35.968968 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8"] Nov 03 00:35:35.972962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:35.972943 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:35.999528 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:35.999505 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:35.999674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:35.999660 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-metrics-certs\") pod \"maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:35.999719 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:35.999710 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c2b60-932a-49da-9d2a-4337ff396131-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:35.999812 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:35.999798 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbrg2\" (UniqueName: \"kubernetes.io/projected/3d4c2b60-932a-49da-9d2a-4337ff396131-kube-api-access-nbrg2\") pod \"maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:36.026785 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.026748 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f2d039-serving-cert\"" Nov 03 00:35:36.078654 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.078625 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8"] Nov 03 00:35:36.101378 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.101345 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:36.101605 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:35:36.101443 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-serving-cert: secret "maas-api-e2e-ipp-b-f2d039-serving-cert" not found Nov 03 00:35:36.101605 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:35:36.101505 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-maas-api-tls podName:3d4c2b60-932a-49da-9d2a-4337ff396131 nodeName:}" failed. No retries permitted until 2026-11-03 00:35:36.601487033 +0000 UTC m=+1783.552427185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-maas-api-tls") pod "maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" (UID: "3d4c2b60-932a-49da-9d2a-4337ff396131") : secret "maas-api-e2e-ipp-b-f2d039-serving-cert" not found Nov 03 00:35:36.101605 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.101550 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-metrics-certs\") pod \"maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:36.101816 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.101615 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c2b60-932a-49da-9d2a-4337ff396131-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:36.101816 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.101664 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbrg2\" (UniqueName: \"kubernetes.io/projected/3d4c2b60-932a-49da-9d2a-4337ff396131-kube-api-access-nbrg2\") pod \"maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:36.102420 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.102393 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c2b60-932a-49da-9d2a-4337ff396131-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:36.104667 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.104642 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-metrics-certs\") pod \"maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:36.159352 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.159319 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbrg2\" (UniqueName: \"kubernetes.io/projected/3d4c2b60-932a-49da-9d2a-4337ff396131-kube-api-access-nbrg2\") pod \"maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:36.181894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.181866 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:36.349432 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.349384 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:36.509700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.509665 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:36.608303 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.608229 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:36.610724 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.610698 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:36.888863 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:36.888752 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:37.264353 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:37.264330 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8"] Nov 03 00:35:37.266955 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:35:37.266924 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4c2b60_932a_49da_9d2a_4337ff396131.slice/crio-73cd62d495d96ab19fdd02ab898cea6bc75d8343a917a5942687214c56416b27 WatchSource:0}: Error finding container 73cd62d495d96ab19fdd02ab898cea6bc75d8343a917a5942687214c56416b27: Status 404 returned error can't find the container with id 73cd62d495d96ab19fdd02ab898cea6bc75d8343a917a5942687214c56416b27 Nov 03 00:35:37.662926 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:37.662896 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" event={"ID":"3d4c2b60-932a-49da-9d2a-4337ff396131","Type":"ContainerStarted","Data":"73cd62d495d96ab19fdd02ab898cea6bc75d8343a917a5942687214c56416b27"} Nov 03 00:35:38.669712 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:38.669676 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" event={"ID":"3d4c2b60-932a-49da-9d2a-4337ff396131","Type":"ContainerStarted","Data":"1295e13118c22328799e45c3d94f7730e2e381bd2ee00462c7f254c1d1c22145"} Nov 03 00:35:38.670116 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:38.669766 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:38.785087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:38.785032 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" podStartSLOduration=2.889866219 podStartE2EDuration="3.785016978s" totalImagesPullingTime="895.150759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:38.78290843 +0000 UTC m=+1785.733848587" watchObservedRunningTime="2026-11-03 00:35:38.785016978 +0000 UTC m=+1785.735957137" Nov 03 00:35:41.205530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:41.205501 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:44.681240 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:44.681207 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:35:46.135768 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:46.135733 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:49.800393 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:49.800345 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:50.588263 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:50.588229 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:50.803186 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:50.803153 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:51.238048 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:51.238013 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:51.576760 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:51.576721 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:56.214738 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:56.214698 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:57.155804 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:35:57.155768 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:35:57.519013 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:35:57.518935 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:00.243139 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:00.243103 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:00.459137 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:00.459111 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:01.541923 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:01.541837 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:05.031370 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:05.031336 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:05.568875 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:05.568171 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:08.294640 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:08.294600 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:08.662606 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:08.662521 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:13.053594 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:13.053561 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:36:14.244051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.244017 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn"] Nov 03 00:36:14.247629 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.247613 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:14.300260 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.300229 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-cb486a-serving-cert\"" Nov 03 00:36:14.352202 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.352178 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn"] Nov 03 00:36:14.356573 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.356548 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94d9s\" (UniqueName: \"kubernetes.io/projected/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-kube-api-access-94d9s\") pod \"maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:14.356698 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.356609 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:14.356891 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.356862 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:14.356986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.356967 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-metrics-certs\") pod \"maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:14.452581 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.452546 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:14.457722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.457697 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:14.457810 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.457752 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-metrics-certs\") pod \"maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:14.457871 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.457812 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94d9s\" (UniqueName: \"kubernetes.io/projected/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-kube-api-access-94d9s\") pod \"maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:14.457871 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:36:14.457856 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-serving-cert: secret "maas-api-e2e-praxis-a-cb486a-serving-cert" not found Nov 03 00:36:14.457943 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.457885 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:14.457943 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:36:14.457912 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-maas-api-tls podName:e8e9bfae-6228-44d2-b1ec-e64ffbf6b105 nodeName:}" failed. No retries permitted until 2026-11-03 00:36:14.957894525 +0000 UTC m=+1821.908834664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-maas-api-tls") pod "maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" (UID: "e8e9bfae-6228-44d2-b1ec-e64ffbf6b105") : secret "maas-api-e2e-praxis-a-cb486a-serving-cert" not found Nov 03 00:36:14.458534 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.458505 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:14.460222 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.460202 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-metrics-certs\") pod \"maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:14.552687 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.552644 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94d9s\" (UniqueName: \"kubernetes.io/projected/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-kube-api-access-94d9s\") pod \"maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:14.622974 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.622943 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:14.777938 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.777893 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:14.967307 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.967231 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:14.969934 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:14.969890 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:15.159908 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:15.159882 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:15.540583 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:15.540556 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn"] Nov 03 00:36:15.542537 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:36:15.542512 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e9bfae_6228_44d2_b1ec_e64ffbf6b105.slice/crio-dcc5cf2ad221293187b48f7923316d291a06dbae430c0f0fed9c54b06e7bbbb7 WatchSource:0}: Error finding container dcc5cf2ad221293187b48f7923316d291a06dbae430c0f0fed9c54b06e7bbbb7: Status 404 returned error can't find the container with id dcc5cf2ad221293187b48f7923316d291a06dbae430c0f0fed9c54b06e7bbbb7 Nov 03 00:36:15.911161 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:15.911090 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" event={"ID":"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105","Type":"ContainerStarted","Data":"dcc5cf2ad221293187b48f7923316d291a06dbae430c0f0fed9c54b06e7bbbb7"} Nov 03 00:36:16.918451 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:16.918417 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" event={"ID":"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105","Type":"ContainerStarted","Data":"b31082ac0ecfd316f8c29e712bfd6204db5eb97ff994ab3d00f4c97819ba9ead"} Nov 03 00:36:16.918851 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:16.918520 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:17.033297 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:17.033243 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" podStartSLOduration=2.25235454 podStartE2EDuration="3.033228169s" totalImagesPullingTime="780.873629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:17.031813995 +0000 UTC m=+1823.982754157" watchObservedRunningTime="2026-11-03 00:36:17.033228169 +0000 UTC m=+1823.984168334" Nov 03 00:36:18.870624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:18.870586 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:19.400222 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:19.400179 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:19.572277 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:19.572231 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:20.358809 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:20.358775 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:20.660521 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:20.660378 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:21.030887 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:21.026471 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:21.410930 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:21.410885 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:22.930316 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:22.930287 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:36:25.388680 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:25.388642 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:25.570073 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:25.570041 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:25.734390 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:25.734301 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:27.155784 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:36:27.155747 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:36:28.802337 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:28.801553 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:28.966510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:28.966476 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:29.656217 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:29.656186 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:29.848313 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:29.848266 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:31.155285 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:31.155244 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:31.367549 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:31.367071 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:31.789423 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:31.789388 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:32.625734 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:32.625697 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:35.063213 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:35.063175 6400 generic.go:356] "Generic (PLEG): container finished" podID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerID="aa42b5d0eb09dc1a0e6155b9ff15a59b978afdd8297081b8665cd0ee3cfce650" exitCode=0 Nov 03 00:36:35.063660 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:35.063247 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" event={"ID":"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53","Type":"ContainerDied","Data":"aa42b5d0eb09dc1a0e6155b9ff15a59b978afdd8297081b8665cd0ee3cfce650"} Nov 03 00:36:35.063660 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:35.063635 6400 scope.go:122] "RemoveContainer" containerID="aa42b5d0eb09dc1a0e6155b9ff15a59b978afdd8297081b8665cd0ee3cfce650" Nov 03 00:36:35.065057 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:35.065028 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:36:35.065171 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:35.065110 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:36:35.065171 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:35.065160 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:36:35.517245 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:35.517169 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:35.905847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:35.902168 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:36.102568 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:36.102524 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" event={"ID":"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53","Type":"ContainerStarted","Data":"5d09bf8826c372ae6570466a0bd7132bced9964a2e09da499d3007a936e984d6"} Nov 03 00:36:36.430227 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:36.430188 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:36:36.431524 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:36.431500 6400 patch_prober.go:29] interesting pod/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:36:36.431654 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:36.431548 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" podUID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 00:36:37.429523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:37.429490 6400 patch_prober.go:29] interesting pod/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:36:37.429938 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:37.429545 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" podUID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 00:36:38.429903 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:38.429861 6400 patch_prober.go:29] interesting pod/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:36:38.430396 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:38.429922 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" podUID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 00:36:38.457290 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:38.457250 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:36:38.461572 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:38.461543 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:39.429320 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:39.429288 6400 patch_prober.go:29] interesting pod/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:36:39.429498 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:39.429339 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" podUID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 00:36:40.096568 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.096524 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh"] Nov 03 00:36:40.115846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.097363 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" podUID="deea825b-3387-4fae-bacf-88afa43dbae4" containerName="maas-api" containerID="cri-o://b7f4daafe337012d180026039fe0b3ece57fc6246f724f2b6abce567ab3375ee" gracePeriod=30 Nov 03 00:36:40.116173 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.112935 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:40.275084 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.274988 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:40.445867 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.430191 6400 patch_prober.go:29] interesting pod/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:36:40.445867 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.430250 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" podUID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 00:36:40.644211 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.644120 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:36:40.657918 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.657878 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-metrics-certs\") pod \"deea825b-3387-4fae-bacf-88afa43dbae4\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " Nov 03 00:36:40.658119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.657939 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-maas-api-tls\") pod \"deea825b-3387-4fae-bacf-88afa43dbae4\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " Nov 03 00:36:40.658119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.657999 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deea825b-3387-4fae-bacf-88afa43dbae4-kube-api-access-252tr\" (UniqueName: \"kubernetes.io/projected/deea825b-3387-4fae-bacf-88afa43dbae4-kube-api-access-252tr\") pod \"deea825b-3387-4fae-bacf-88afa43dbae4\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " Nov 03 00:36:40.658119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.658082 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deea825b-3387-4fae-bacf-88afa43dbae4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deea825b-3387-4fae-bacf-88afa43dbae4-openshift-service-ca\") pod \"deea825b-3387-4fae-bacf-88afa43dbae4\" (UID: \"deea825b-3387-4fae-bacf-88afa43dbae4\") " Nov 03 00:36:40.659606 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.659574 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deea825b-3387-4fae-bacf-88afa43dbae4-openshift-service-ca" pod "deea825b-3387-4fae-bacf-88afa43dbae4" (UID: "deea825b-3387-4fae-bacf-88afa43dbae4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:40.661443 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.661404 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deea825b-3387-4fae-bacf-88afa43dbae4-kube-api-access-252tr" pod "deea825b-3387-4fae-bacf-88afa43dbae4" (UID: "deea825b-3387-4fae-bacf-88afa43dbae4"). InnerVolumeSpecName "kube-api-access-252tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:40.662437 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.662413 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-maas-api-tls" pod "deea825b-3387-4fae-bacf-88afa43dbae4" (UID: "deea825b-3387-4fae-bacf-88afa43dbae4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:36:40.662502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.662453 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-metrics-certs" pod "deea825b-3387-4fae-bacf-88afa43dbae4" (UID: "deea825b-3387-4fae-bacf-88afa43dbae4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:36:40.759287 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.759254 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:40.759287 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.759282 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deea825b-3387-4fae-bacf-88afa43dbae4-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:40.759287 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.759293 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-252tr\" (UniqueName: \"kubernetes.io/projected/deea825b-3387-4fae-bacf-88afa43dbae4-kube-api-access-252tr\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:40.759524 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:40.759304 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deea825b-3387-4fae-bacf-88afa43dbae4-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:41.167850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:41.167103 6400 generic.go:356] "Generic (PLEG): container finished" podID="deea825b-3387-4fae-bacf-88afa43dbae4" containerID="b7f4daafe337012d180026039fe0b3ece57fc6246f724f2b6abce567ab3375ee" exitCode=0 Nov 03 00:36:41.167850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:41.167196 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" event={"ID":"deea825b-3387-4fae-bacf-88afa43dbae4","Type":"ContainerDied","Data":"b7f4daafe337012d180026039fe0b3ece57fc6246f724f2b6abce567ab3375ee"} Nov 03 00:36:41.167850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:41.167242 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" event={"ID":"deea825b-3387-4fae-bacf-88afa43dbae4","Type":"ContainerDied","Data":"8ad7f916db6487c79416a6cc65d8bab816e0239d5f1a21923611d0a66a3464bb"} Nov 03 00:36:41.167850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:41.167270 6400 scope.go:122] "RemoveContainer" containerID="b7f4daafe337012d180026039fe0b3ece57fc6246f724f2b6abce567ab3375ee" Nov 03 00:36:41.167850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:41.167581 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh" Nov 03 00:36:41.204962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:41.204873 6400 scope.go:122] "RemoveContainer" containerID="b7f4daafe337012d180026039fe0b3ece57fc6246f724f2b6abce567ab3375ee" Nov 03 00:36:41.205874 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:36:41.205803 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f4daafe337012d180026039fe0b3ece57fc6246f724f2b6abce567ab3375ee\": container with ID starting with b7f4daafe337012d180026039fe0b3ece57fc6246f724f2b6abce567ab3375ee not found: ID does not exist" containerID="b7f4daafe337012d180026039fe0b3ece57fc6246f724f2b6abce567ab3375ee" Nov 03 00:36:41.205999 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:41.205968 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f4daafe337012d180026039fe0b3ece57fc6246f724f2b6abce567ab3375ee"} err="failed to get container status \"b7f4daafe337012d180026039fe0b3ece57fc6246f724f2b6abce567ab3375ee\": rpc error: code = NotFound desc = could not find container \"b7f4daafe337012d180026039fe0b3ece57fc6246f724f2b6abce567ab3375ee\": container with ID starting with b7f4daafe337012d180026039fe0b3ece57fc6246f724f2b6abce567ab3375ee not found: ID does not exist" Nov 03 00:36:41.262160 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:36:41.262124 6400 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeea825b_3387_4fae_bacf_88afa43dbae4.slice\": RecentStats: unable to find data in memory cache]" Nov 03 00:36:41.343087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:41.343054 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh"] Nov 03 00:36:41.346237 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:41.346212 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb20df-66bf7c8b54-z9drh"] Nov 03 00:36:41.434305 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:41.434221 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:36:41.435272 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:41.435245 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:36:42.115868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:42.115812 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:42.271332 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:42.271300 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:43.179934 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:43.179900 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deea825b-3387-4fae-bacf-88afa43dbae4" path="/var/lib/kubelet/pods/deea825b-3387-4fae-bacf-88afa43dbae4/volumes" Nov 03 00:36:43.217991 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:43.217964 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:44.431499 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:44.431466 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:44.679070 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:44.679041 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:45.071328 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:45.071293 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:45.093897 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:45.093866 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:36:45.410021 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:45.409938 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:45.572718 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:45.572683 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:45.888872 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:45.888829 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:46.378533 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:46.378493 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz"] Nov 03 00:36:46.378761 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:46.378733 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerName="istio-proxy" containerID="cri-o://935d03468dec41fa8aca4ff048ce65e47a6618662c76e1f8daf49d22e9b10917" gracePeriod=30 Nov 03 00:36:46.516503 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:46.516465 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:47.341515 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:47.341481 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:48.512364 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:48.512333 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:49.058463 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:49.058431 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:49.679879 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:49.679795 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:50.242292 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:50.242262 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:51.267336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.267287 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:51.649182 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.649149 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:36:51.726096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.726057 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-data\") pod \"1e679802-f6b3-49e5-9c52-f523957ce3ed\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " Nov 03 00:36:51.726279 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.726131 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-certs\") pod \"1e679802-f6b3-49e5-9c52-f523957ce3ed\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " Nov 03 00:36:51.726279 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.726153 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-credential-socket\") pod \"1e679802-f6b3-49e5-9c52-f523957ce3ed\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " Nov 03 00:36:51.726279 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.726256 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-envoy\") pod \"1e679802-f6b3-49e5-9c52-f523957ce3ed\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " Nov 03 00:36:51.726462 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.726282 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-socket\") pod \"1e679802-f6b3-49e5-9c52-f523957ce3ed\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " Nov 03 00:36:51.726462 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.726345 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e679802-f6b3-49e5-9c52-f523957ce3ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e679802-f6b3-49e5-9c52-f523957ce3ed-istiod-ca-cert\") pod \"1e679802-f6b3-49e5-9c52-f523957ce3ed\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " Nov 03 00:36:51.726462 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.726390 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-kube-api-access-x4pjq\" (UniqueName: \"kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-kube-api-access-x4pjq\") pod \"1e679802-f6b3-49e5-9c52-f523957ce3ed\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " Nov 03 00:36:51.726462 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.726428 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-data" pod "1e679802-f6b3-49e5-9c52-f523957ce3ed" (UID: "1e679802-f6b3-49e5-9c52-f523957ce3ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:51.726462 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.726452 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-token\" (UniqueName: \"kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-token\") pod \"1e679802-f6b3-49e5-9c52-f523957ce3ed\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " Nov 03 00:36:51.726710 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.726447 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-certs" pod "1e679802-f6b3-49e5-9c52-f523957ce3ed" (UID: "1e679802-f6b3-49e5-9c52-f523957ce3ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:51.726710 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.726541 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-podinfo\") pod \"1e679802-f6b3-49e5-9c52-f523957ce3ed\" (UID: \"1e679802-f6b3-49e5-9c52-f523957ce3ed\") " Nov 03 00:36:51.726872 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.726745 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e679802-f6b3-49e5-9c52-f523957ce3ed-istiod-ca-cert" pod "1e679802-f6b3-49e5-9c52-f523957ce3ed" (UID: "1e679802-f6b3-49e5-9c52-f523957ce3ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:51.727353 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.727321 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-credential-socket" pod "1e679802-f6b3-49e5-9c52-f523957ce3ed" (UID: "1e679802-f6b3-49e5-9c52-f523957ce3ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:51.727353 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.727335 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:51.727550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.727376 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:51.727550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.727418 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e679802-f6b3-49e5-9c52-f523957ce3ed-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:51.727550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.727425 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-socket" pod "1e679802-f6b3-49e5-9c52-f523957ce3ed" (UID: "1e679802-f6b3-49e5-9c52-f523957ce3ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:51.728766 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.728742 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-envoy" pod "1e679802-f6b3-49e5-9c52-f523957ce3ed" (UID: "1e679802-f6b3-49e5-9c52-f523957ce3ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:51.728922 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.728895 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-podinfo" pod "1e679802-f6b3-49e5-9c52-f523957ce3ed" (UID: "1e679802-f6b3-49e5-9c52-f523957ce3ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:36:51.729020 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.728919 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-kube-api-access-x4pjq" pod "1e679802-f6b3-49e5-9c52-f523957ce3ed" (UID: "1e679802-f6b3-49e5-9c52-f523957ce3ed"). InnerVolumeSpecName "kube-api-access-x4pjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:51.729020 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.728928 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-token" pod "1e679802-f6b3-49e5-9c52-f523957ce3ed" (UID: "1e679802-f6b3-49e5-9c52-f523957ce3ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:51.829062 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.828982 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:51.829062 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.829009 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:51.829062 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.829022 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4pjq\" (UniqueName: \"kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-kube-api-access-x4pjq\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:51.829062 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.829035 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:51.829062 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.829047 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e679802-f6b3-49e5-9c52-f523957ce3ed-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:51.829062 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:51.829059 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e679802-f6b3-49e5-9c52-f523957ce3ed-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:52.244008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:52.243905 6400 generic.go:356] "Generic (PLEG): container finished" podID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerID="935d03468dec41fa8aca4ff048ce65e47a6618662c76e1f8daf49d22e9b10917" exitCode=0 Nov 03 00:36:52.244008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:52.243961 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" Nov 03 00:36:52.244008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:52.243989 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" event={"ID":"1e679802-f6b3-49e5-9c52-f523957ce3ed","Type":"ContainerDied","Data":"935d03468dec41fa8aca4ff048ce65e47a6618662c76e1f8daf49d22e9b10917"} Nov 03 00:36:52.244279 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:52.244056 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz" event={"ID":"1e679802-f6b3-49e5-9c52-f523957ce3ed","Type":"ContainerDied","Data":"a9edd3a42c365e548774bd0aa42d1a603de55507de0c8a531063d08bc915425d"} Nov 03 00:36:52.244279 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:52.244088 6400 scope.go:122] "RemoveContainer" containerID="935d03468dec41fa8aca4ff048ce65e47a6618662c76e1f8daf49d22e9b10917" Nov 03 00:36:52.254862 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:52.254843 6400 scope.go:122] "RemoveContainer" containerID="b94b032c32511accf166739d2f73eaa6d6b8ab2e7489de9e4364df3923116859" Nov 03 00:36:52.264016 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:52.263996 6400 scope.go:122] "RemoveContainer" containerID="935d03468dec41fa8aca4ff048ce65e47a6618662c76e1f8daf49d22e9b10917" Nov 03 00:36:52.264330 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:36:52.264301 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935d03468dec41fa8aca4ff048ce65e47a6618662c76e1f8daf49d22e9b10917\": container with ID starting with 935d03468dec41fa8aca4ff048ce65e47a6618662c76e1f8daf49d22e9b10917 not found: ID does not exist" containerID="935d03468dec41fa8aca4ff048ce65e47a6618662c76e1f8daf49d22e9b10917" Nov 03 00:36:52.264451 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:52.264338 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935d03468dec41fa8aca4ff048ce65e47a6618662c76e1f8daf49d22e9b10917"} err="failed to get container status \"935d03468dec41fa8aca4ff048ce65e47a6618662c76e1f8daf49d22e9b10917\": rpc error: code = NotFound desc = could not find container \"935d03468dec41fa8aca4ff048ce65e47a6618662c76e1f8daf49d22e9b10917\": container with ID starting with 935d03468dec41fa8aca4ff048ce65e47a6618662c76e1f8daf49d22e9b10917 not found: ID does not exist" Nov 03 00:36:52.264451 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:52.264356 6400 scope.go:122] "RemoveContainer" containerID="b94b032c32511accf166739d2f73eaa6d6b8ab2e7489de9e4364df3923116859" Nov 03 00:36:52.264631 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:36:52.264610 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94b032c32511accf166739d2f73eaa6d6b8ab2e7489de9e4364df3923116859\": container with ID starting with b94b032c32511accf166739d2f73eaa6d6b8ab2e7489de9e4364df3923116859 not found: ID does not exist" containerID="b94b032c32511accf166739d2f73eaa6d6b8ab2e7489de9e4364df3923116859" Nov 03 00:36:52.264681 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:52.264637 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94b032c32511accf166739d2f73eaa6d6b8ab2e7489de9e4364df3923116859"} err="failed to get container status \"b94b032c32511accf166739d2f73eaa6d6b8ab2e7489de9e4364df3923116859\": rpc error: code = NotFound desc = could not find container \"b94b032c32511accf166739d2f73eaa6d6b8ab2e7489de9e4364df3923116859\": container with ID starting with b94b032c32511accf166739d2f73eaa6d6b8ab2e7489de9e4364df3923116859 not found: ID does not exist" Nov 03 00:36:52.411432 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:52.411403 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz"] Nov 03 00:36:52.461308 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:52.461279 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb20df-openshift-default-67c5885fbc-lpfdz"] Nov 03 00:36:52.526011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:52.525923 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:53.179425 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:53.179395 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" path="/var/lib/kubelet/pods/1e679802-f6b3-49e5-9c52-f523957ce3ed/volumes" Nov 03 00:36:53.626412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:53.626378 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:53.818722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:53.818692 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:54.000277 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:54.000184 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:56.856528 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:56.856497 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:57.065179 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:57.065138 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:57.200518 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:36:57.200439 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:36:57.793504 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:36:57.793461 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:03.387717 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.387677 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4"] Nov 03 00:37:03.388254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.387932 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" podUID="2b82a6a0-d03f-4a10-af02-a0770917e943" containerName="maas-api" containerID="cri-o://daa1ea0896aa8112d8fdfa972d4f7d392852560e4e5d0933a02eac693933feee" gracePeriod=30 Nov 03 00:37:03.514162 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.514130 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:03.721286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.721259 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:37:03.785849 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.779736 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b82a6a0-d03f-4a10-af02-a0770917e943-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b82a6a0-d03f-4a10-af02-a0770917e943-openshift-service-ca\") pod \"2b82a6a0-d03f-4a10-af02-a0770917e943\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " Nov 03 00:37:03.798847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.794556 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b82a6a0-d03f-4a10-af02-a0770917e943-kube-api-access-fv59m\" (UniqueName: \"kubernetes.io/projected/2b82a6a0-d03f-4a10-af02-a0770917e943-kube-api-access-fv59m\") pod \"2b82a6a0-d03f-4a10-af02-a0770917e943\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " Nov 03 00:37:03.798847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.794665 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-maas-api-tls\") pod \"2b82a6a0-d03f-4a10-af02-a0770917e943\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " Nov 03 00:37:03.798847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.794760 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-metrics-certs\") pod \"2b82a6a0-d03f-4a10-af02-a0770917e943\" (UID: \"2b82a6a0-d03f-4a10-af02-a0770917e943\") " Nov 03 00:37:03.814398 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.793647 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b82a6a0-d03f-4a10-af02-a0770917e943-openshift-service-ca" pod "2b82a6a0-d03f-4a10-af02-a0770917e943" (UID: "2b82a6a0-d03f-4a10-af02-a0770917e943"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:37:03.822499 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.822464 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b82a6a0-d03f-4a10-af02-a0770917e943-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:03.824661 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.824629 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b82a6a0-d03f-4a10-af02-a0770917e943-kube-api-access-fv59m" pod "2b82a6a0-d03f-4a10-af02-a0770917e943" (UID: "2b82a6a0-d03f-4a10-af02-a0770917e943"). InnerVolumeSpecName "kube-api-access-fv59m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:37:03.842172 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.842137 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-maas-api-tls" pod "2b82a6a0-d03f-4a10-af02-a0770917e943" (UID: "2b82a6a0-d03f-4a10-af02-a0770917e943"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:37:03.844770 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.844696 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-metrics-certs" pod "2b82a6a0-d03f-4a10-af02-a0770917e943" (UID: "2b82a6a0-d03f-4a10-af02-a0770917e943"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:37:03.923927 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.923895 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:03.923927 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.923921 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b82a6a0-d03f-4a10-af02-a0770917e943-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:03.923927 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:03.923934 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv59m\" (UniqueName: \"kubernetes.io/projected/2b82a6a0-d03f-4a10-af02-a0770917e943-kube-api-access-fv59m\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:04.050762 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:04.050728 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:04.247781 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:04.247737 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:04.340169 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:04.340081 6400 generic.go:356] "Generic (PLEG): container finished" podID="2b82a6a0-d03f-4a10-af02-a0770917e943" containerID="daa1ea0896aa8112d8fdfa972d4f7d392852560e4e5d0933a02eac693933feee" exitCode=0 Nov 03 00:37:04.340169 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:04.340142 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" Nov 03 00:37:04.340402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:04.340163 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" event={"ID":"2b82a6a0-d03f-4a10-af02-a0770917e943","Type":"ContainerDied","Data":"daa1ea0896aa8112d8fdfa972d4f7d392852560e4e5d0933a02eac693933feee"} Nov 03 00:37:04.340402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:04.340257 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4" event={"ID":"2b82a6a0-d03f-4a10-af02-a0770917e943","Type":"ContainerDied","Data":"bc5c5ad02224c2291a0a084bd17940b870645081057105a077df77714d620d64"} Nov 03 00:37:04.340402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:04.340287 6400 scope.go:122] "RemoveContainer" containerID="daa1ea0896aa8112d8fdfa972d4f7d392852560e4e5d0933a02eac693933feee" Nov 03 00:37:04.351080 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:04.351056 6400 scope.go:122] "RemoveContainer" containerID="daa1ea0896aa8112d8fdfa972d4f7d392852560e4e5d0933a02eac693933feee" Nov 03 00:37:04.351377 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:37:04.351358 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa1ea0896aa8112d8fdfa972d4f7d392852560e4e5d0933a02eac693933feee\": container with ID starting with daa1ea0896aa8112d8fdfa972d4f7d392852560e4e5d0933a02eac693933feee not found: ID does not exist" containerID="daa1ea0896aa8112d8fdfa972d4f7d392852560e4e5d0933a02eac693933feee" Nov 03 00:37:04.351447 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:04.351389 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa1ea0896aa8112d8fdfa972d4f7d392852560e4e5d0933a02eac693933feee"} err="failed to get container status \"daa1ea0896aa8112d8fdfa972d4f7d392852560e4e5d0933a02eac693933feee\": rpc error: code = NotFound desc = could not find container \"daa1ea0896aa8112d8fdfa972d4f7d392852560e4e5d0933a02eac693933feee\": container with ID starting with daa1ea0896aa8112d8fdfa972d4f7d392852560e4e5d0933a02eac693933feee not found: ID does not exist" Nov 03 00:37:04.503542 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:04.503511 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4"] Nov 03 00:37:04.552756 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:04.552729 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3df441-54549fbc96-6j2q4"] Nov 03 00:37:05.180289 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:05.180257 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b82a6a0-d03f-4a10-af02-a0770917e943" path="/var/lib/kubelet/pods/2b82a6a0-d03f-4a10-af02-a0770917e943/volumes" Nov 03 00:37:05.313402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:05.313368 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:05.537230 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:05.537200 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:07.043776 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:07.043741 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:07.530356 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:07.530273 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:08.313928 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:08.313899 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:09.004432 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:09.004400 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:10.499612 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:10.499572 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:12.938447 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:12.938414 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:14.058130 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:14.058079 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:14.219870 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:14.219810 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:14.921891 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:14.902394 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:15.722841 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:15.722794 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:15.897353 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:15.897322 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:16.385839 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:16.385782 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:16.728653 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:16.728562 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:19.836097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:19.836063 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:20.027331 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:20.027293 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:20.297961 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:20.297923 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:21.109595 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:21.109559 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:24.064435 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:24.064400 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:24.259573 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:24.259541 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:26.340647 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:26.340604 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:27.153805 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:37:27.153766 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:37:30.487567 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:30.487539 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8"] Nov 03 00:37:30.487984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:30.487732 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" podUID="3d4c2b60-932a-49da-9d2a-4337ff396131" containerName="maas-api" containerID="cri-o://1295e13118c22328799e45c3d94f7730e2e381bd2ee00462c7f254c1d1c22145" gracePeriod=30 Nov 03 00:37:30.614752 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:30.614719 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:30.811577 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:30.811549 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:37:30.951690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:30.951660 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-maas-api-tls\") pod \"3d4c2b60-932a-49da-9d2a-4337ff396131\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " Nov 03 00:37:30.951911 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:30.951709 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4c2b60-932a-49da-9d2a-4337ff396131-kube-api-access-nbrg2\" (UniqueName: \"kubernetes.io/projected/3d4c2b60-932a-49da-9d2a-4337ff396131-kube-api-access-nbrg2\") pod \"3d4c2b60-932a-49da-9d2a-4337ff396131\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " Nov 03 00:37:30.951911 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:30.951760 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4c2b60-932a-49da-9d2a-4337ff396131-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c2b60-932a-49da-9d2a-4337ff396131-openshift-service-ca\") pod \"3d4c2b60-932a-49da-9d2a-4337ff396131\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " Nov 03 00:37:30.951911 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:30.951869 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-metrics-certs\") pod \"3d4c2b60-932a-49da-9d2a-4337ff396131\" (UID: \"3d4c2b60-932a-49da-9d2a-4337ff396131\") " Nov 03 00:37:30.952452 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:30.952422 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4c2b60-932a-49da-9d2a-4337ff396131-openshift-service-ca" pod "3d4c2b60-932a-49da-9d2a-4337ff396131" (UID: "3d4c2b60-932a-49da-9d2a-4337ff396131"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:37:30.954557 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:30.954527 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4c2b60-932a-49da-9d2a-4337ff396131-kube-api-access-nbrg2" pod "3d4c2b60-932a-49da-9d2a-4337ff396131" (UID: "3d4c2b60-932a-49da-9d2a-4337ff396131"). InnerVolumeSpecName "kube-api-access-nbrg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:37:30.954719 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:30.954534 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-maas-api-tls" pod "3d4c2b60-932a-49da-9d2a-4337ff396131" (UID: "3d4c2b60-932a-49da-9d2a-4337ff396131"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:37:30.956103 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:30.956075 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-metrics-certs" pod "3d4c2b60-932a-49da-9d2a-4337ff396131" (UID: "3d4c2b60-932a-49da-9d2a-4337ff396131"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:37:31.057003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:31.056895 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c2b60-932a-49da-9d2a-4337ff396131-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:31.057257 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:31.057236 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:31.057354 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:31.057343 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4c2b60-932a-49da-9d2a-4337ff396131-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:31.057437 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:31.057423 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbrg2\" (UniqueName: \"kubernetes.io/projected/3d4c2b60-932a-49da-9d2a-4337ff396131-kube-api-access-nbrg2\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:31.549414 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:31.549385 6400 generic.go:356] "Generic (PLEG): container finished" podID="3d4c2b60-932a-49da-9d2a-4337ff396131" containerID="1295e13118c22328799e45c3d94f7730e2e381bd2ee00462c7f254c1d1c22145" exitCode=0 Nov 03 00:37:31.549797 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:31.549431 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" Nov 03 00:37:31.549797 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:31.549469 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" event={"ID":"3d4c2b60-932a-49da-9d2a-4337ff396131","Type":"ContainerDied","Data":"1295e13118c22328799e45c3d94f7730e2e381bd2ee00462c7f254c1d1c22145"} Nov 03 00:37:31.549797 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:31.549529 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8" event={"ID":"3d4c2b60-932a-49da-9d2a-4337ff396131","Type":"ContainerDied","Data":"73cd62d495d96ab19fdd02ab898cea6bc75d8343a917a5942687214c56416b27"} Nov 03 00:37:31.549797 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:31.549558 6400 scope.go:122] "RemoveContainer" containerID="1295e13118c22328799e45c3d94f7730e2e381bd2ee00462c7f254c1d1c22145" Nov 03 00:37:31.558973 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:31.558954 6400 scope.go:122] "RemoveContainer" containerID="1295e13118c22328799e45c3d94f7730e2e381bd2ee00462c7f254c1d1c22145" Nov 03 00:37:31.559239 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:37:31.559219 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1295e13118c22328799e45c3d94f7730e2e381bd2ee00462c7f254c1d1c22145\": container with ID starting with 1295e13118c22328799e45c3d94f7730e2e381bd2ee00462c7f254c1d1c22145 not found: ID does not exist" containerID="1295e13118c22328799e45c3d94f7730e2e381bd2ee00462c7f254c1d1c22145" Nov 03 00:37:31.559304 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:31.559252 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1295e13118c22328799e45c3d94f7730e2e381bd2ee00462c7f254c1d1c22145"} err="failed to get container status \"1295e13118c22328799e45c3d94f7730e2e381bd2ee00462c7f254c1d1c22145\": rpc error: code = NotFound desc = could not find container \"1295e13118c22328799e45c3d94f7730e2e381bd2ee00462c7f254c1d1c22145\": container with ID starting with 1295e13118c22328799e45c3d94f7730e2e381bd2ee00462c7f254c1d1c22145 not found: ID does not exist" Nov 03 00:37:31.712056 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:31.712028 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8"] Nov 03 00:37:31.761577 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:31.761551 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2d039-7885cc8685-kbrv8"] Nov 03 00:37:32.439494 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:32.439450 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:33.178521 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:33.178489 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4c2b60-932a-49da-9d2a-4337ff396131" path="/var/lib/kubelet/pods/3d4c2b60-932a-49da-9d2a-4337ff396131/volumes" Nov 03 00:37:37.096595 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:37.096565 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:37.454001 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:37.453911 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:38.646296 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:38.646267 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:39.127433 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:39.127400 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:40.551026 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:40.550980 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:40.980240 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:40.980152 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:37:41.030506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.030465 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx"] Nov 03 00:37:41.031013 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.030995 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerName="istio-proxy" Nov 03 00:37:41.031128 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.031014 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerName="istio-proxy" Nov 03 00:37:41.031128 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.031077 6400 cpu_manager.go:414] "containerMap: removing container" podUID="deea825b-3387-4fae-bacf-88afa43dbae4" containerName="maas-api" Nov 03 00:37:41.031128 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.031086 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deea825b-3387-4fae-bacf-88afa43dbae4" containerName="maas-api" Nov 03 00:37:41.031128 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.031103 6400 cpu_manager.go:414] "containerMap: removing container" podUID="2b82a6a0-d03f-4a10-af02-a0770917e943" containerName="maas-api" Nov 03 00:37:41.031128 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.031110 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b82a6a0-d03f-4a10-af02-a0770917e943" containerName="maas-api" Nov 03 00:37:41.031128 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.031124 6400 cpu_manager.go:414] "containerMap: removing container" podUID="3d4c2b60-932a-49da-9d2a-4337ff396131" containerName="maas-api" Nov 03 00:37:41.031412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.031131 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4c2b60-932a-49da-9d2a-4337ff396131" containerName="maas-api" Nov 03 00:37:41.031412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.031255 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerName="istio-proxy" Nov 03 00:37:41.031412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.031276 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b82a6a0-d03f-4a10-af02-a0770917e943" containerName="maas-api" Nov 03 00:37:41.031412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.031285 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="deea825b-3387-4fae-bacf-88afa43dbae4" containerName="maas-api" Nov 03 00:37:41.031412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.031295 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4c2b60-932a-49da-9d2a-4337ff396131" containerName="maas-api" Nov 03 00:37:41.034166 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.034146 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.059589 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.059534 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-token\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.059764 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.059703 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f4272a-6597-4d6a-9f23-8562ba6944cd-istiod-ca-cert\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.059878 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.059857 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n45cd\" (UniqueName: \"kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-kube-api-access-n45cd\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.059947 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.059915 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-socket\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.060036 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.060008 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-envoy\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.060201 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.060174 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-podinfo\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.060331 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.060233 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-credential-socket\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.060402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.060338 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-certs\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.060402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.060387 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-data\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.086159 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.086131 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d61eae-openshift-default-dockercfg-ms8qk\"" Nov 03 00:37:41.112687 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.112661 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx"] Nov 03 00:37:41.154906 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.154869 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:41.161224 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.161204 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-token\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.161298 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.161256 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f4272a-6597-4d6a-9f23-8562ba6944cd-istiod-ca-cert\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.161298 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.161285 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n45cd\" (UniqueName: \"kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-kube-api-access-n45cd\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.161402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.161327 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-socket\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.161402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.161396 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-envoy\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.161500 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.161446 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-podinfo\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.161500 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.161489 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-credential-socket\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.161612 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.161547 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-certs\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.161612 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.161573 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-data\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.162078 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.162052 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f4272a-6597-4d6a-9f23-8562ba6944cd-istiod-ca-cert\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.162197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.162170 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-socket\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.162325 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.162259 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-credential-socket\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.162385 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.162321 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-certs\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.162385 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.162351 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-data\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.164270 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.164240 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-envoy\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.164424 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.164406 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-podinfo\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.218682 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.218648 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-token\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.219068 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.219043 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n45cd\" (UniqueName: \"kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-kube-api-access-n45cd\") pod \"e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.331910 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.331865 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:41.349685 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.349647 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:41.782136 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.782053 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx"] Nov 03 00:37:41.822082 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:37:41.822055 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f4272a_6597_4d6a_9f23_8562ba6944cd.slice/crio-3a4179cd40abda52b0c8dde2ca5b06c27c115734178499860ce7857224753702 WatchSource:0}: Error finding container 3a4179cd40abda52b0c8dde2ca5b06c27c115734178499860ce7857224753702: Status 404 returned error can't find the container with id 3a4179cd40abda52b0c8dde2ca5b06c27c115734178499860ce7857224753702 Nov 03 00:37:41.824841 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.824778 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:37:41.824947 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.824896 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:37:41.824947 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.824930 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:37:41.836347 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.836325 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj"] Nov 03 00:37:41.836963 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.836942 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerName="istio-proxy" Nov 03 00:37:41.837069 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.836976 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerName="istio-proxy" Nov 03 00:37:41.837233 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.837217 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e679802-f6b3-49e5-9c52-f523957ce3ed" containerName="istio-proxy" Nov 03 00:37:41.840256 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.840237 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:41.868910 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.868888 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:41.869005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.868927 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-metrics-certs\") pod \"maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:41.869067 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.869045 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qxjk\" (UniqueName: \"kubernetes.io/projected/1c854675-3a17-4ead-ac15-3364f3de99f5-kube-api-access-4qxjk\") pod \"maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:41.869195 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.869177 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c854675-3a17-4ead-ac15-3364f3de99f5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:41.895728 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.895701 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-75dfee-serving-cert\"" Nov 03 00:37:41.899190 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.899161 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj"] Nov 03 00:37:41.970209 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.970172 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qxjk\" (UniqueName: \"kubernetes.io/projected/1c854675-3a17-4ead-ac15-3364f3de99f5-kube-api-access-4qxjk\") pod \"maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:41.970393 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.970279 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c854675-3a17-4ead-ac15-3364f3de99f5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:41.970479 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.970455 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:41.970627 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.970492 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-metrics-certs\") pod \"maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:41.970806 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:37:41.970783 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-serving-cert: secret "maas-api-e2e-praxis-a-75dfee-serving-cert" not found Nov 03 00:37:41.970906 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:37:41.970890 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-maas-api-tls podName:1c854675-3a17-4ead-ac15-3364f3de99f5 nodeName:}" failed. No retries permitted until 2026-11-03 00:37:42.470870176 +0000 UTC m=+1909.421810315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-maas-api-tls") pod "maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" (UID: "1c854675-3a17-4ead-ac15-3364f3de99f5") : secret "maas-api-e2e-praxis-a-75dfee-serving-cert" not found Nov 03 00:37:41.971213 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.971166 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c854675-3a17-4ead-ac15-3364f3de99f5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:41.973992 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:41.973968 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-metrics-certs\") pod \"maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:42.029909 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:42.029884 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qxjk\" (UniqueName: \"kubernetes.io/projected/1c854675-3a17-4ead-ac15-3364f3de99f5-kube-api-access-4qxjk\") pod \"maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:42.056619 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:42.056588 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:42.272809 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:42.272760 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:42.438705 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:42.438599 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:42.477565 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:42.477537 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:42.480188 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:42.480161 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:42.610791 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:42.610758 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" event={"ID":"22f4272a-6597-4d6a-9f23-8562ba6944cd","Type":"ContainerStarted","Data":"cf3940076e8293246d5c5f2c9210bd140ef0cf1d83a40a23adb5ecf32142d063"} Nov 03 00:37:42.611002 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:42.610816 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" event={"ID":"22f4272a-6597-4d6a-9f23-8562ba6944cd","Type":"ContainerStarted","Data":"3a4179cd40abda52b0c8dde2ca5b06c27c115734178499860ce7857224753702"} Nov 03 00:37:42.728295 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:42.728202 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" podStartSLOduration=2.72818799 podStartE2EDuration="2.72818799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:42.72592509 +0000 UTC m=+1909.676865249" watchObservedRunningTime="2026-11-03 00:37:42.72818799 +0000 UTC m=+1909.679128147" Nov 03 00:37:42.770725 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:42.770699 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:43.145384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:43.145358 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj"] Nov 03 00:37:43.147140 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:37:43.147109 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c854675_3a17_4ead_ac15_3364f3de99f5.slice/crio-1b5519430de4c5bff68b42a67642212b189d4d5159e0205c336e592ef503cf87 WatchSource:0}: Error finding container 1b5519430de4c5bff68b42a67642212b189d4d5159e0205c336e592ef503cf87: Status 404 returned error can't find the container with id 1b5519430de4c5bff68b42a67642212b189d4d5159e0205c336e592ef503cf87 Nov 03 00:37:43.148748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:43.148728 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:37:43.350548 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:43.350519 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:43.355783 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:43.355756 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:43.617424 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:43.617388 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" event={"ID":"1c854675-3a17-4ead-ac15-3364f3de99f5","Type":"ContainerStarted","Data":"1b5519430de4c5bff68b42a67642212b189d4d5159e0205c336e592ef503cf87"} Nov 03 00:37:43.617785 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:43.617766 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:43.618848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:43.618802 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:37:44.624421 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:44.624384 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" event={"ID":"1c854675-3a17-4ead-ac15-3364f3de99f5","Type":"ContainerStarted","Data":"a7738e1a0f58290bcc98f8cb2c62b9a096dea5fa3a18ac4fa6e060f05ae32689"} Nov 03 00:37:44.624774 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:44.624552 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:44.738588 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:44.738527 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" podStartSLOduration=2.870916742 podStartE2EDuration="3.738510946s" totalImagesPullingTime="867.594204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:44.738051589 +0000 UTC m=+1911.688991748" watchObservedRunningTime="2026-11-03 00:37:44.738510946 +0000 UTC m=+1911.689451145" Nov 03 00:37:45.553427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:45.553397 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:45.749998 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:45.749966 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:45.931485 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:45.931391 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:47.341650 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:47.341605 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:50.372999 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:50.372959 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:50.571590 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:50.571555 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:50.683564 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:50.683476 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:37:52.665072 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:52.665033 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:57.151925 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:37:57.151880 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:37:59.042723 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:37:59.042689 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:03.705907 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:03.705855 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:04.643362 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:04.643332 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:38:07.020943 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:07.020904 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:08.560342 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:08.556262 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:09.352476 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:09.352436 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:12.868496 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:12.868464 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq"] Nov 03 00:38:12.872125 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:12.872106 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:12.926150 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:12.926109 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-0db2fb-serving-cert\"" Nov 03 00:38:12.934936 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:12.934912 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xg2g\" (UniqueName: \"kubernetes.io/projected/b16a4886-e342-4469-a1eb-77a97f8ad0d9-kube-api-access-5xg2g\") pod \"maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:12.935059 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:12.934961 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16a4886-e342-4469-a1eb-77a97f8ad0d9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:12.935103 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:12.935080 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-metrics-certs\") pod \"maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:12.935144 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:12.935123 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:12.977446 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:12.977419 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq"] Nov 03 00:38:13.035668 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.035630 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16a4886-e342-4469-a1eb-77a97f8ad0d9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:13.035833 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.035729 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-metrics-certs\") pod \"maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:13.035833 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.035774 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:13.035965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.035840 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xg2g\" (UniqueName: \"kubernetes.io/projected/b16a4886-e342-4469-a1eb-77a97f8ad0d9-kube-api-access-5xg2g\") pod \"maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:13.035965 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:38:13.035931 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-serving-cert: secret "maas-api-e2e-praxis-b-0db2fb-serving-cert" not found Nov 03 00:38:13.036079 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:38:13.035998 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-maas-api-tls podName:b16a4886-e342-4469-a1eb-77a97f8ad0d9 nodeName:}" failed. No retries permitted until 2026-11-03 00:38:13.535978006 +0000 UTC m=+1940.486918158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-maas-api-tls") pod "maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" (UID: "b16a4886-e342-4469-a1eb-77a97f8ad0d9") : secret "maas-api-e2e-praxis-b-0db2fb-serving-cert" not found Nov 03 00:38:13.036434 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.036413 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16a4886-e342-4469-a1eb-77a97f8ad0d9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:13.038453 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.038431 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-metrics-certs\") pod \"maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:13.078522 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.078494 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:13.094805 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.094777 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xg2g\" (UniqueName: \"kubernetes.io/projected/b16a4886-e342-4469-a1eb-77a97f8ad0d9-kube-api-access-5xg2g\") pod \"maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:13.306968 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.306921 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:13.466334 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.466293 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:13.543107 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.543076 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:13.545660 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.545631 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:13.737430 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.737351 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:13.784757 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:13.784730 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:14.158392 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:14.158360 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq"] Nov 03 00:38:14.160363 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:38:14.160332 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16a4886_e342_4469_a1eb_77a97f8ad0d9.slice/crio-675b10ff01e9883629ab255bcba39cf4e544f05ace7dbffe15d29c9c4bef61ea WatchSource:0}: Error finding container 675b10ff01e9883629ab255bcba39cf4e544f05ace7dbffe15d29c9c4bef61ea: Status 404 returned error can't find the container with id 675b10ff01e9883629ab255bcba39cf4e544f05ace7dbffe15d29c9c4bef61ea Nov 03 00:38:14.804844 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:14.804271 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" event={"ID":"b16a4886-e342-4469-a1eb-77a97f8ad0d9","Type":"ContainerStarted","Data":"675b10ff01e9883629ab255bcba39cf4e544f05ace7dbffe15d29c9c4bef61ea"} Nov 03 00:38:14.914445 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:14.914119 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:15.328145 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:15.328114 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:15.811523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:15.811484 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" event={"ID":"b16a4886-e342-4469-a1eb-77a97f8ad0d9","Type":"ContainerStarted","Data":"d3723b50a3500123ed77fa77e27bb2984550d4efc146d34a689ab821ae3507af"} Nov 03 00:38:15.811693 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:15.811550 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:15.925139 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:15.925084 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" podStartSLOduration=3.008488716 podStartE2EDuration="3.925070048s" totalImagesPullingTime="916.581332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:15.924333196 +0000 UTC m=+1942.875273355" watchObservedRunningTime="2026-11-03 00:38:15.925070048 +0000 UTC m=+1942.876010239" Nov 03 00:38:17.849958 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:17.849927 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:18.587495 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:18.587462 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:19.928508 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:19.928479 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:20.158479 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:20.158447 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:21.822536 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:21.822510 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:38:22.530956 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:22.530907 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:22.708417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:22.707326 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:23.854309 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:23.854274 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:26.092895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.092864 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts"] Nov 03 00:38:26.096997 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.096979 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.120735 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.120709 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:26.155952 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.155929 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts"] Nov 03 00:38:26.186224 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.186191 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-model-cache\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.186365 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.186234 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-home\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.186434 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.186374 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wljf\" (UniqueName: \"kubernetes.io/projected/af260334-9c1b-43e8-a923-ed50a40ecce8-kube-api-access-5wljf\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.186487 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.186429 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-tmp-dir\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.186534 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.186517 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-dshm\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.186578 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.186566 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af260334-9c1b-43e8-a923-ed50a40ecce8-tls-certs\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.203217 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.203190 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1a883c\"/\"openshift-service-ca.crt\"" Nov 03 00:38:26.203346 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.203271 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1a883c\"/\"kube-root-ca.crt\"" Nov 03 00:38:26.203346 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.203290 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1a883c\"/\"default-dockercfg-b4lp5\"" Nov 03 00:38:26.207410 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.203546 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1a883c\"/\"ipp-route-1a883c-kserve-self-signed-certs\"" Nov 03 00:38:26.288089 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.288061 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-model-cache\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.288248 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.288116 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-home\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.288248 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.288168 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wljf\" (UniqueName: \"kubernetes.io/projected/af260334-9c1b-43e8-a923-ed50a40ecce8-kube-api-access-5wljf\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.288248 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.288194 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-tmp-dir\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.288446 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.288415 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-dshm\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.288543 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.288518 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af260334-9c1b-43e8-a923-ed50a40ecce8-tls-certs\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.288669 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.288568 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-model-cache\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.288669 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.288619 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-home\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.288761 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.288737 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-tmp-dir\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.290857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.290835 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-dshm\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.290998 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.290983 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af260334-9c1b-43e8-a923-ed50a40ecce8-tls-certs\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.320278 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.320252 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:26.350042 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.349993 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wljf\" (UniqueName: \"kubernetes.io/projected/af260334-9c1b-43e8-a923-ed50a40ecce8-kube-api-access-5wljf\") pod \"ipp-route-1a883c-kserve-f84f4484f-lk8ts\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.409692 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.409665 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:26.492860 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.488100 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:26.791718 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.791689 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts"] Nov 03 00:38:26.792339 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:38:26.792316 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf260334_9c1b_43e8_a923_ed50a40ecce8.slice/crio-469062fab935e8aa83ec85f891cde3a11f4fd73255cf9ecc2da2ab81bd253c6e WatchSource:0}: Error finding container 469062fab935e8aa83ec85f891cde3a11f4fd73255cf9ecc2da2ab81bd253c6e: Status 404 returned error can't find the container with id 469062fab935e8aa83ec85f891cde3a11f4fd73255cf9ecc2da2ab81bd253c6e Nov 03 00:38:26.869991 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.869958 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" event={"ID":"af260334-9c1b-43e8-a923-ed50a40ecce8","Type":"ContainerStarted","Data":"7615964ee0bd0a744b970f36803e7e2bc270fc9e6e050906bf5c97feca609331"} Nov 03 00:38:26.870092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.870031 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" event={"ID":"af260334-9c1b-43e8-a923-ed50a40ecce8","Type":"ContainerStarted","Data":"469062fab935e8aa83ec85f891cde3a11f4fd73255cf9ecc2da2ab81bd253c6e"} Nov 03 00:38:26.992676 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:26.992626 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" podStartSLOduration=0.992611519 podStartE2EDuration="992.611519ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:26.991781545 +0000 UTC m=+1953.942721704" watchObservedRunningTime="2026-11-03 00:38:26.992611519 +0000 UTC m=+1953.943551677" Nov 03 00:38:27.171264 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:38:27.171183 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:38:27.410298 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:27.410256 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:28.621387 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:28.621357 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:33.313119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:33.313083 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:36.410704 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:36.410674 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:36.422631 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:36.422603 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:36.942698 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:36.942668 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:38:38.029053 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:38.029010 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:38.438739 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:38.438707 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:38:42.580079 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:42.580039 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:42.767068 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:42.766988 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:42.921959 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:42.921881 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:45.809287 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:45.809257 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:46.994968 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:46.994939 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:47.601091 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:47.601061 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:47.807318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:47.807283 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:47.966189 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:47.966115 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:51.752332 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:51.752297 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:51.921010 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:51.920972 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:52.088168 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:52.088136 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:52.931964 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:52.931929 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:55.362584 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:55.362543 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:55.552243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:55.552209 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:57.157169 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:38:57.157123 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:38:59.604597 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:38:59.604560 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:04.315222 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:04.315141 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:09.293218 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:09.293182 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:11.178028 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:11.177987 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:12.794586 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:12.794528 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:13.297100 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:13.297052 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:13.648070 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:13.648036 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc"] Nov 03 00:39:13.650052 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:13.648236 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" podUID="699aded9-a381-4ee0-820e-3c568b5a4ad7" containerName="istio-proxy" containerID="cri-o://c1a390a4b784185a0a924fb7e767011467c274e606ab4542e1344e8c698ec79b" gracePeriod=30 Nov 03 00:39:13.661972 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:13.661853 6400 patch_prober.go:29] interesting pod/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": read tcp 10.134.0.2:42424->10.134.0.88:15021: read: connection reset by peer" start-of-body= Nov 03 00:39:13.661972 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:13.661907 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": read tcp 10.134.0.2:42424->10.134.0.88:15021: read: connection reset by peer" Nov 03 00:39:13.670315 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:13.670286 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z"] Nov 03 00:39:13.670528 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:13.670505 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" podUID="42973673-96bf-4c81-9b1d-570456790ee3" containerName="istio-proxy" containerID="cri-o://9aef367b02ce5292c2b51208d6b50210ad87edeb3147c0eedb2fecfd9ca0ee1f" gracePeriod=30 Nov 03 00:39:14.162472 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:14.162440 6400 generic.go:356] "Generic (PLEG): container finished" podID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerID="cf3940076e8293246d5c5f2c9210bd140ef0cf1d83a40a23adb5ecf32142d063" exitCode=0 Nov 03 00:39:14.162875 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:14.162518 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" event={"ID":"22f4272a-6597-4d6a-9f23-8562ba6944cd","Type":"ContainerDied","Data":"cf3940076e8293246d5c5f2c9210bd140ef0cf1d83a40a23adb5ecf32142d063"} Nov 03 00:39:14.162964 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:14.162949 6400 scope.go:122] "RemoveContainer" containerID="cf3940076e8293246d5c5f2c9210bd140ef0cf1d83a40a23adb5ecf32142d063" Nov 03 00:39:14.164448 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:14.164419 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:39:14.164536 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:14.164490 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:39:14.164536 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:14.164521 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:39:15.181454 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:15.181407 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" event={"ID":"22f4272a-6597-4d6a-9f23-8562ba6944cd","Type":"ContainerStarted","Data":"ccabfa5d6733f38ecd5ed18c4ddacfa340647b2f0b4397006452d660626f3bd4"} Nov 03 00:39:15.350688 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:15.350654 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:39:15.350883 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:15.350792 6400 patch_prober.go:29] interesting pod/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 00:39:15.350883 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:15.350861 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 00:39:15.934841 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:15.934792 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:16.350137 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:16.350098 6400 patch_prober.go:29] interesting pod/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 00:39:16.350525 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:16.350148 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 00:39:17.350708 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:17.350672 6400 patch_prober.go:29] interesting pod/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 00:39:17.351156 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:17.350729 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 00:39:18.350187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:18.350153 6400 patch_prober.go:29] interesting pod/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 00:39:18.350373 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:18.350204 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 00:39:18.406613 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:18.406581 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:39:18.407011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:18.406635 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" podUID="699aded9-a381-4ee0-820e-3c568b5a4ad7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:39:19.014087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.014060 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:39:19.035782 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.035760 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:39:19.108553 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.108518 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-data\") pod \"699aded9-a381-4ee0-820e-3c568b5a4ad7\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " Nov 03 00:39:19.108752 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.108605 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-socket\") pod \"42973673-96bf-4c81-9b1d-570456790ee3\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " Nov 03 00:39:19.109214 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109186 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-credential-socket\") pod \"42973673-96bf-4c81-9b1d-570456790ee3\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " Nov 03 00:39:19.109214 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109218 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-data" pod "699aded9-a381-4ee0-820e-3c568b5a4ad7" (UID: "699aded9-a381-4ee0-820e-3c568b5a4ad7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:19.109386 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109246 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699aded9-a381-4ee0-820e-3c568b5a4ad7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699aded9-a381-4ee0-820e-3c568b5a4ad7-istiod-ca-cert\") pod \"699aded9-a381-4ee0-820e-3c568b5a4ad7\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " Nov 03 00:39:19.109386 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109357 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-envoy\") pod \"699aded9-a381-4ee0-820e-3c568b5a4ad7\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " Nov 03 00:39:19.109472 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109392 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-token\" (UniqueName: \"kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-token\") pod \"699aded9-a381-4ee0-820e-3c568b5a4ad7\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " Nov 03 00:39:19.109472 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109413 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-socket" pod "42973673-96bf-4c81-9b1d-570456790ee3" (UID: "42973673-96bf-4c81-9b1d-570456790ee3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:19.109571 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109475 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-podinfo\") pod \"699aded9-a381-4ee0-820e-3c568b5a4ad7\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " Nov 03 00:39:19.109571 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109525 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-kube-api-access-pgt89\" (UniqueName: \"kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-kube-api-access-pgt89\") pod \"42973673-96bf-4c81-9b1d-570456790ee3\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " Nov 03 00:39:19.109571 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109560 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-envoy\") pod \"42973673-96bf-4c81-9b1d-570456790ee3\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " Nov 03 00:39:19.109721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109597 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-data\") pod \"42973673-96bf-4c81-9b1d-570456790ee3\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " Nov 03 00:39:19.109721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109641 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42973673-96bf-4c81-9b1d-570456790ee3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42973673-96bf-4c81-9b1d-570456790ee3-istio-podinfo\") pod \"42973673-96bf-4c81-9b1d-570456790ee3\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " Nov 03 00:39:19.109721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109675 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-certs\") pod \"42973673-96bf-4c81-9b1d-570456790ee3\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " Nov 03 00:39:19.109894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109722 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-certs\") pod \"699aded9-a381-4ee0-820e-3c568b5a4ad7\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " Nov 03 00:39:19.109894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109755 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-kube-api-access-2s48l\" (UniqueName: \"kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-kube-api-access-2s48l\") pod \"699aded9-a381-4ee0-820e-3c568b5a4ad7\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " Nov 03 00:39:19.109894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109758 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-credential-socket" pod "42973673-96bf-4c81-9b1d-570456790ee3" (UID: "42973673-96bf-4c81-9b1d-570456790ee3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:19.109894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109772 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699aded9-a381-4ee0-820e-3c568b5a4ad7-istiod-ca-cert" pod "699aded9-a381-4ee0-820e-3c568b5a4ad7" (UID: "699aded9-a381-4ee0-820e-3c568b5a4ad7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:39:19.109894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109794 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42973673-96bf-4c81-9b1d-570456790ee3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42973673-96bf-4c81-9b1d-570456790ee3-istiod-ca-cert\") pod \"42973673-96bf-4c81-9b1d-570456790ee3\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " Nov 03 00:39:19.109894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109890 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-socket\") pod \"699aded9-a381-4ee0-820e-3c568b5a4ad7\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " Nov 03 00:39:19.110166 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109922 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-credential-socket\") pod \"699aded9-a381-4ee0-820e-3c568b5a4ad7\" (UID: \"699aded9-a381-4ee0-820e-3c568b5a4ad7\") " Nov 03 00:39:19.110166 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109963 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-istio-token\" (UniqueName: \"kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-istio-token\") pod \"42973673-96bf-4c81-9b1d-570456790ee3\" (UID: \"42973673-96bf-4c81-9b1d-570456790ee3\") " Nov 03 00:39:19.110166 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.109992 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-certs" pod "699aded9-a381-4ee0-820e-3c568b5a4ad7" (UID: "699aded9-a381-4ee0-820e-3c568b5a4ad7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:19.110166 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.110157 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-certs" pod "42973673-96bf-4c81-9b1d-570456790ee3" (UID: "42973673-96bf-4c81-9b1d-570456790ee3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:19.110504 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.110312 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-socket" pod "699aded9-a381-4ee0-820e-3c568b5a4ad7" (UID: "699aded9-a381-4ee0-820e-3c568b5a4ad7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:19.110504 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.110477 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.110504 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.110496 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.110642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.110510 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.110642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.110523 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.110642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.110535 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.110642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.110550 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.110642 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.110562 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699aded9-a381-4ee0-820e-3c568b5a4ad7-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.110918 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.110729 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-credential-socket" pod "699aded9-a381-4ee0-820e-3c568b5a4ad7" (UID: "699aded9-a381-4ee0-820e-3c568b5a4ad7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:19.110918 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.110867 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42973673-96bf-4c81-9b1d-570456790ee3-istiod-ca-cert" pod "42973673-96bf-4c81-9b1d-570456790ee3" (UID: "42973673-96bf-4c81-9b1d-570456790ee3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:39:19.111628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.111366 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-data" pod "42973673-96bf-4c81-9b1d-570456790ee3" (UID: "42973673-96bf-4c81-9b1d-570456790ee3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:19.114552 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.114520 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-podinfo" pod "699aded9-a381-4ee0-820e-3c568b5a4ad7" (UID: "699aded9-a381-4ee0-820e-3c568b5a4ad7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:39:19.114552 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.114538 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-istio-token" pod "42973673-96bf-4c81-9b1d-570456790ee3" (UID: "42973673-96bf-4c81-9b1d-570456790ee3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:19.114684 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.114580 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-envoy" pod "699aded9-a381-4ee0-820e-3c568b5a4ad7" (UID: "699aded9-a381-4ee0-820e-3c568b5a4ad7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:19.114684 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.114636 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-kube-api-access-2s48l" pod "699aded9-a381-4ee0-820e-3c568b5a4ad7" (UID: "699aded9-a381-4ee0-820e-3c568b5a4ad7"). InnerVolumeSpecName "kube-api-access-2s48l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:19.114684 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.114636 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-token" pod "699aded9-a381-4ee0-820e-3c568b5a4ad7" (UID: "699aded9-a381-4ee0-820e-3c568b5a4ad7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:19.114890 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.114868 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-envoy" pod "42973673-96bf-4c81-9b1d-570456790ee3" (UID: "42973673-96bf-4c81-9b1d-570456790ee3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:19.115276 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.115255 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42973673-96bf-4c81-9b1d-570456790ee3-istio-podinfo" pod "42973673-96bf-4c81-9b1d-570456790ee3" (UID: "42973673-96bf-4c81-9b1d-570456790ee3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:39:19.115496 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.115478 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-kube-api-access-pgt89" pod "42973673-96bf-4c81-9b1d-570456790ee3" (UID: "42973673-96bf-4c81-9b1d-570456790ee3"). InnerVolumeSpecName "kube-api-access-pgt89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:19.211502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.211411 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.211502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.211441 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42973673-96bf-4c81-9b1d-570456790ee3-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.211502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.211451 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42973673-96bf-4c81-9b1d-570456790ee3-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.211502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.211461 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s48l\" (UniqueName: \"kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-kube-api-access-2s48l\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.211502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.211471 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42973673-96bf-4c81-9b1d-570456790ee3-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.211502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.211479 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.211502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.211488 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.211502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.211496 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.211502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.211503 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.211502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.211511 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699aded9-a381-4ee0-820e-3c568b5a4ad7-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.212135 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.211520 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgt89\" (UniqueName: \"kubernetes.io/projected/42973673-96bf-4c81-9b1d-570456790ee3-kube-api-access-pgt89\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:19.228994 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.228964 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:39:19.234487 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.234454 6400 generic.go:356] "Generic (PLEG): container finished" podID="699aded9-a381-4ee0-820e-3c568b5a4ad7" containerID="c1a390a4b784185a0a924fb7e767011467c274e606ab4542e1344e8c698ec79b" exitCode=0 Nov 03 00:39:19.234641 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.234561 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" Nov 03 00:39:19.234641 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.234582 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" event={"ID":"699aded9-a381-4ee0-820e-3c568b5a4ad7","Type":"ContainerDied","Data":"c1a390a4b784185a0a924fb7e767011467c274e606ab4542e1344e8c698ec79b"} Nov 03 00:39:19.234641 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.234633 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc" event={"ID":"699aded9-a381-4ee0-820e-3c568b5a4ad7","Type":"ContainerDied","Data":"8b6674402970b0a27db7e427aac900cf27b91e6016ce59bbb7717290770ddcaf"} Nov 03 00:39:19.234789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.234654 6400 scope.go:122] "RemoveContainer" containerID="c1a390a4b784185a0a924fb7e767011467c274e606ab4542e1344e8c698ec79b" Nov 03 00:39:19.236869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.236265 6400 generic.go:356] "Generic (PLEG): container finished" podID="42973673-96bf-4c81-9b1d-570456790ee3" containerID="9aef367b02ce5292c2b51208d6b50210ad87edeb3147c0eedb2fecfd9ca0ee1f" exitCode=0 Nov 03 00:39:19.236869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.236333 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" event={"ID":"42973673-96bf-4c81-9b1d-570456790ee3","Type":"ContainerDied","Data":"9aef367b02ce5292c2b51208d6b50210ad87edeb3147c0eedb2fecfd9ca0ee1f"} Nov 03 00:39:19.236869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.236348 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" Nov 03 00:39:19.236869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.236371 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z" event={"ID":"42973673-96bf-4c81-9b1d-570456790ee3","Type":"ContainerDied","Data":"ab6598fd6516f37afe1828c3e0757f5bffb046268947b7c889487b7cae470c0d"} Nov 03 00:39:19.244401 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.244381 6400 scope.go:122] "RemoveContainer" containerID="c1a390a4b784185a0a924fb7e767011467c274e606ab4542e1344e8c698ec79b" Nov 03 00:39:19.244680 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:39:19.244660 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a390a4b784185a0a924fb7e767011467c274e606ab4542e1344e8c698ec79b\": container with ID starting with c1a390a4b784185a0a924fb7e767011467c274e606ab4542e1344e8c698ec79b not found: ID does not exist" containerID="c1a390a4b784185a0a924fb7e767011467c274e606ab4542e1344e8c698ec79b" Nov 03 00:39:19.244734 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.244702 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a390a4b784185a0a924fb7e767011467c274e606ab4542e1344e8c698ec79b"} err="failed to get container status \"c1a390a4b784185a0a924fb7e767011467c274e606ab4542e1344e8c698ec79b\": rpc error: code = NotFound desc = could not find container \"c1a390a4b784185a0a924fb7e767011467c274e606ab4542e1344e8c698ec79b\": container with ID starting with c1a390a4b784185a0a924fb7e767011467c274e606ab4542e1344e8c698ec79b not found: ID does not exist" Nov 03 00:39:19.244734 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.244720 6400 scope.go:122] "RemoveContainer" containerID="9aef367b02ce5292c2b51208d6b50210ad87edeb3147c0eedb2fecfd9ca0ee1f" Nov 03 00:39:19.253129 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.253114 6400 scope.go:122] "RemoveContainer" containerID="9aef367b02ce5292c2b51208d6b50210ad87edeb3147c0eedb2fecfd9ca0ee1f" Nov 03 00:39:19.253362 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:39:19.253348 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aef367b02ce5292c2b51208d6b50210ad87edeb3147c0eedb2fecfd9ca0ee1f\": container with ID starting with 9aef367b02ce5292c2b51208d6b50210ad87edeb3147c0eedb2fecfd9ca0ee1f not found: ID does not exist" containerID="9aef367b02ce5292c2b51208d6b50210ad87edeb3147c0eedb2fecfd9ca0ee1f" Nov 03 00:39:19.253421 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.253372 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aef367b02ce5292c2b51208d6b50210ad87edeb3147c0eedb2fecfd9ca0ee1f"} err="failed to get container status \"9aef367b02ce5292c2b51208d6b50210ad87edeb3147c0eedb2fecfd9ca0ee1f\": rpc error: code = NotFound desc = could not find container \"9aef367b02ce5292c2b51208d6b50210ad87edeb3147c0eedb2fecfd9ca0ee1f\": container with ID starting with 9aef367b02ce5292c2b51208d6b50210ad87edeb3147c0eedb2fecfd9ca0ee1f not found: ID does not exist" Nov 03 00:39:19.350981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.350947 6400 patch_prober.go:29] interesting pod/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 00:39:19.351150 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.350999 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 00:39:19.437875 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.437788 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc"] Nov 03 00:39:19.440094 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.440069 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-wzwgc"] Nov 03 00:39:19.562921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.562888 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z"] Nov 03 00:39:19.564739 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:19.564712 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-85t8z"] Nov 03 00:39:20.350980 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:20.350948 6400 patch_prober.go:29] interesting pod/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 00:39:20.351175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:20.351041 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 00:39:20.713258 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:20.713168 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:20.920076 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:20.920035 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:21.145039 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:21.145003 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:21.179477 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:21.179430 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42973673-96bf-4c81-9b1d-570456790ee3" path="/var/lib/kubelet/pods/42973673-96bf-4c81-9b1d-570456790ee3/volumes" Nov 03 00:39:21.180011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:21.179991 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="699aded9-a381-4ee0-820e-3c568b5a4ad7" path="/var/lib/kubelet/pods/699aded9-a381-4ee0-820e-3c568b5a4ad7/volumes" Nov 03 00:39:21.354944 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:21.354916 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:39:21.355196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:21.355180 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:39:21.355900 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:21.355882 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:39:22.193651 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:22.193612 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:25.755978 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:25.755931 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:25.964274 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:25.964245 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:27.156891 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:39:27.156851 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:39:30.552291 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:30.552252 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:31.677308 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:31.677271 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:32.365062 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:32.365030 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:32.531283 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:32.531246 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:35.622949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:35.622914 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:37.249948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:37.249907 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:40.412523 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:40.412490 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:45.116211 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:45.116174 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:49.750598 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:49.750563 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:54.446815 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:54.446782 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:57.152069 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:39:57.152034 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:39:58.350178 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:58.350120 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:59.450261 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:59.450233 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:59.613722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:59.613685 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:59.769723 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:39:59.769641 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:00.162979 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:00.162943 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:00.426359 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:00.426272 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:04.949096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:04.949062 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:05.156896 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:05.156857 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:05.367750 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:05.367717 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:05.547977 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:05.547945 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:08.419254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.419216 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:08.622267 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.622233 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:08.816154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.816124 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476"] Nov 03 00:40:08.816618 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.816606 6400 cpu_manager.go:414] "containerMap: removing container" podUID="42973673-96bf-4c81-9b1d-570456790ee3" containerName="istio-proxy" Nov 03 00:40:08.816660 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.816619 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42973673-96bf-4c81-9b1d-570456790ee3" containerName="istio-proxy" Nov 03 00:40:08.816660 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.816630 6400 cpu_manager.go:414] "containerMap: removing container" podUID="699aded9-a381-4ee0-820e-3c568b5a4ad7" containerName="istio-proxy" Nov 03 00:40:08.816660 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.816636 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699aded9-a381-4ee0-820e-3c568b5a4ad7" containerName="istio-proxy" Nov 03 00:40:08.816755 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.816711 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="42973673-96bf-4c81-9b1d-570456790ee3" containerName="istio-proxy" Nov 03 00:40:08.816755 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.816721 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="699aded9-a381-4ee0-820e-3c568b5a4ad7" containerName="istio-proxy" Nov 03 00:40:08.821321 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.821302 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" Nov 03 00:40:08.860295 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.860259 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34111ab-2498-45bc-bfad-f35a49cffd7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476\" (UID: \"c34111ab-2498-45bc-bfad-f35a49cffd7f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" Nov 03 00:40:08.860476 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.860327 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgptl\" (UniqueName: \"kubernetes.io/projected/c34111ab-2498-45bc-bfad-f35a49cffd7f-kube-api-access-cgptl\") pod \"payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476\" (UID: \"c34111ab-2498-45bc-bfad-f35a49cffd7f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" Nov 03 00:40:08.875537 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.875508 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476"] Nov 03 00:40:08.922910 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.922376 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-bdd194\"" Nov 03 00:40:08.922910 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.922537 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-bdd194-dockercfg-sp6j5\"" Nov 03 00:40:08.964948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.962524 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgptl\" (UniqueName: \"kubernetes.io/projected/c34111ab-2498-45bc-bfad-f35a49cffd7f-kube-api-access-cgptl\") pod \"payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476\" (UID: \"c34111ab-2498-45bc-bfad-f35a49cffd7f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" Nov 03 00:40:08.965219 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.965193 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34111ab-2498-45bc-bfad-f35a49cffd7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476\" (UID: \"c34111ab-2498-45bc-bfad-f35a49cffd7f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" Nov 03 00:40:08.966096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:08.966069 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34111ab-2498-45bc-bfad-f35a49cffd7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476\" (UID: \"c34111ab-2498-45bc-bfad-f35a49cffd7f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" Nov 03 00:40:09.021233 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:09.021200 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgptl\" (UniqueName: \"kubernetes.io/projected/c34111ab-2498-45bc-bfad-f35a49cffd7f-kube-api-access-cgptl\") pod \"payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476\" (UID: \"c34111ab-2498-45bc-bfad-f35a49cffd7f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" Nov 03 00:40:09.134711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:09.134618 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" Nov 03 00:40:09.510287 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:09.510259 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476"] Nov 03 00:40:09.514133 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:40:09.514107 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34111ab_2498_45bc_bfad_f35a49cffd7f.slice/crio-ad08d6c0c154e840488bd7b5968ed96936dca20d1a2ad29ca291ec62fdddbe40 WatchSource:0}: Error finding container ad08d6c0c154e840488bd7b5968ed96936dca20d1a2ad29ca291ec62fdddbe40: Status 404 returned error can't find the container with id ad08d6c0c154e840488bd7b5968ed96936dca20d1a2ad29ca291ec62fdddbe40 Nov 03 00:40:09.544790 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:09.544750 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" event={"ID":"c34111ab-2498-45bc-bfad-f35a49cffd7f","Type":"ContainerStarted","Data":"ad08d6c0c154e840488bd7b5968ed96936dca20d1a2ad29ca291ec62fdddbe40"} Nov 03 00:40:10.047259 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:10.047224 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:10.189898 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:10.189864 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:40:10.850900 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:10.850867 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn"] Nov 03 00:40:10.851376 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:10.851107 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" podUID="e8e9bfae-6228-44d2-b1ec-e64ffbf6b105" containerName="maas-api" containerID="cri-o://b31082ac0ecfd316f8c29e712bfd6204db5eb97ff994ab3d00f4c97819ba9ead" gracePeriod=30 Nov 03 00:40:10.972525 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:10.972491 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:11.283586 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.283543 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:40:11.400504 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.400472 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-maas-api-tls\") pod \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " Nov 03 00:40:11.400696 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.400552 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-openshift-service-ca\") pod \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " Nov 03 00:40:11.400696 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.400593 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-metrics-certs\") pod \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " Nov 03 00:40:11.400696 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.400683 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-kube-api-access-94d9s\" (UniqueName: \"kubernetes.io/projected/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-kube-api-access-94d9s\") pod \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\" (UID: \"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105\") " Nov 03 00:40:11.400981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.400955 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-openshift-service-ca" pod "e8e9bfae-6228-44d2-b1ec-e64ffbf6b105" (UID: "e8e9bfae-6228-44d2-b1ec-e64ffbf6b105"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:40:11.401182 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.401164 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:11.403038 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.403009 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-metrics-certs" pod "e8e9bfae-6228-44d2-b1ec-e64ffbf6b105" (UID: "e8e9bfae-6228-44d2-b1ec-e64ffbf6b105"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:40:11.403416 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.403393 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-kube-api-access-94d9s" pod "e8e9bfae-6228-44d2-b1ec-e64ffbf6b105" (UID: "e8e9bfae-6228-44d2-b1ec-e64ffbf6b105"). InnerVolumeSpecName "kube-api-access-94d9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:40:11.406084 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.406058 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-maas-api-tls" pod "e8e9bfae-6228-44d2-b1ec-e64ffbf6b105" (UID: "e8e9bfae-6228-44d2-b1ec-e64ffbf6b105"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:40:11.502375 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.502302 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:11.502375 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.502340 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:11.502375 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.502355 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94d9s\" (UniqueName: \"kubernetes.io/projected/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105-kube-api-access-94d9s\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:11.558517 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.558259 6400 generic.go:356] "Generic (PLEG): container finished" podID="e8e9bfae-6228-44d2-b1ec-e64ffbf6b105" containerID="b31082ac0ecfd316f8c29e712bfd6204db5eb97ff994ab3d00f4c97819ba9ead" exitCode=0 Nov 03 00:40:11.558517 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.558324 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" event={"ID":"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105","Type":"ContainerDied","Data":"b31082ac0ecfd316f8c29e712bfd6204db5eb97ff994ab3d00f4c97819ba9ead"} Nov 03 00:40:11.558517 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.558349 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" Nov 03 00:40:11.558517 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.558373 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn" event={"ID":"e8e9bfae-6228-44d2-b1ec-e64ffbf6b105","Type":"ContainerDied","Data":"dcc5cf2ad221293187b48f7923316d291a06dbae430c0f0fed9c54b06e7bbbb7"} Nov 03 00:40:11.558517 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.558401 6400 scope.go:122] "RemoveContainer" containerID="b31082ac0ecfd316f8c29e712bfd6204db5eb97ff994ab3d00f4c97819ba9ead" Nov 03 00:40:11.573831 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.573798 6400 scope.go:122] "RemoveContainer" containerID="b31082ac0ecfd316f8c29e712bfd6204db5eb97ff994ab3d00f4c97819ba9ead" Nov 03 00:40:11.574137 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:40:11.574115 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31082ac0ecfd316f8c29e712bfd6204db5eb97ff994ab3d00f4c97819ba9ead\": container with ID starting with b31082ac0ecfd316f8c29e712bfd6204db5eb97ff994ab3d00f4c97819ba9ead not found: ID does not exist" containerID="b31082ac0ecfd316f8c29e712bfd6204db5eb97ff994ab3d00f4c97819ba9ead" Nov 03 00:40:11.574265 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.574147 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31082ac0ecfd316f8c29e712bfd6204db5eb97ff994ab3d00f4c97819ba9ead"} err="failed to get container status \"b31082ac0ecfd316f8c29e712bfd6204db5eb97ff994ab3d00f4c97819ba9ead\": rpc error: code = NotFound desc = could not find container \"b31082ac0ecfd316f8c29e712bfd6204db5eb97ff994ab3d00f4c97819ba9ead\": container with ID starting with b31082ac0ecfd316f8c29e712bfd6204db5eb97ff994ab3d00f4c97819ba9ead not found: ID does not exist" Nov 03 00:40:11.720099 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.720067 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn"] Nov 03 00:40:11.727095 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:11.727066 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cb486a-5847df756d-xkxmn"] Nov 03 00:40:12.703794 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:12.703761 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:13.179782 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:13.179747 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e9bfae-6228-44d2-b1ec-e64ffbf6b105" path="/var/lib/kubelet/pods/e8e9bfae-6228-44d2-b1ec-e64ffbf6b105/volumes" Nov 03 00:40:13.571773 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:13.571738 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" event={"ID":"c34111ab-2498-45bc-bfad-f35a49cffd7f","Type":"ContainerStarted","Data":"d060ec3e8a9213251fa61f9ff98e1a5eb7e895e25f8b351f43377213dec6269c"} Nov 03 00:40:13.571943 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:13.571845 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" Nov 03 00:40:13.685138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:13.685085 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" podStartSLOduration=2.591285842 podStartE2EDuration="5.68507031s" totalImagesPullingTime="3.093784468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:13.683322882 +0000 UTC m=+2060.634263038" watchObservedRunningTime="2026-11-03 00:40:13.68507031 +0000 UTC m=+2060.636010465" Nov 03 00:40:17.546647 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:17.546617 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:18.135777 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:18.135743 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:18.303543 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:18.303498 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:19.740212 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:19.740183 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:20.235447 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:20.235412 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd"] Nov 03 00:40:20.235649 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:20.235606 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" podUID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerName="istio-proxy" containerID="cri-o://5d09bf8826c372ae6570466a0bd7132bced9964a2e09da499d3007a936e984d6" gracePeriod=30 Nov 03 00:40:20.275735 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:20.275692 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:21.036244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:21.036213 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:21.696466 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:21.696434 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:24.347139 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:24.347099 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:24.579607 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:24.579583 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" Nov 03 00:40:25.496559 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.496497 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:40:25.576556 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.576524 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-podinfo\") pod \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " Nov 03 00:40:25.576713 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.576612 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-credential-socket\") pod \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " Nov 03 00:40:25.576713 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.576631 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-envoy\") pod \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " Nov 03 00:40:25.576713 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.576666 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-certs\") pod \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " Nov 03 00:40:25.576908 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.576718 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-data\") pod \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " Nov 03 00:40:25.576908 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.576742 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istiod-ca-cert\") pod \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " Nov 03 00:40:25.576908 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.576783 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-kube-api-access-cjpp6\" (UniqueName: \"kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-kube-api-access-cjpp6\") pod \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " Nov 03 00:40:25.576908 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.576861 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-socket\") pod \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " Nov 03 00:40:25.577141 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.576935 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-token\" (UniqueName: \"kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-token\") pod \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\" (UID: \"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53\") " Nov 03 00:40:25.577141 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.576941 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-credential-socket" pod "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" (UID: "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:40:25.577141 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.577057 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-data" pod "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" (UID: "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:40:25.577463 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.577170 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-socket" pod "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" (UID: "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:40:25.577463 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.577205 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istiod-ca-cert" pod "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" (UID: "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:40:25.577463 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.577445 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-certs" pod "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" (UID: "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:40:25.577621 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.577469 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:25.577621 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.577491 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:25.577621 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.577507 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:25.577621 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.577521 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:25.578852 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.578804 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-podinfo" pod "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" (UID: "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:40:25.578954 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.578911 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-envoy" pod "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" (UID: "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:40:25.579065 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.579047 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-kube-api-access-cjpp6" pod "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" (UID: "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53"). InnerVolumeSpecName "kube-api-access-cjpp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:40:25.579295 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.579278 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-token" pod "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" (UID: "0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:40:25.648974 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.648946 6400 generic.go:356] "Generic (PLEG): container finished" podID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerID="5d09bf8826c372ae6570466a0bd7132bced9964a2e09da499d3007a936e984d6" exitCode=0 Nov 03 00:40:25.649128 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.649000 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" Nov 03 00:40:25.649128 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.649028 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" event={"ID":"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53","Type":"ContainerDied","Data":"5d09bf8826c372ae6570466a0bd7132bced9964a2e09da499d3007a936e984d6"} Nov 03 00:40:25.649128 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.649085 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd" event={"ID":"0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53","Type":"ContainerDied","Data":"3124b72012d183b8f8b07a5931a06ebbfaf5e6d3a4965157e51da6e8451fd4cc"} Nov 03 00:40:25.649128 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.649107 6400 scope.go:122] "RemoveContainer" containerID="5d09bf8826c372ae6570466a0bd7132bced9964a2e09da499d3007a936e984d6" Nov 03 00:40:25.659020 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.659004 6400 scope.go:122] "RemoveContainer" containerID="aa42b5d0eb09dc1a0e6155b9ff15a59b978afdd8297081b8665cd0ee3cfce650" Nov 03 00:40:25.681367 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.681321 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:25.681367 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.681361 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:25.681593 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.681380 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:25.681593 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.681394 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:25.681593 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.681408 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjpp6\" (UniqueName: \"kubernetes.io/projected/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53-kube-api-access-cjpp6\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:25.684313 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.684292 6400 scope.go:122] "RemoveContainer" containerID="5d09bf8826c372ae6570466a0bd7132bced9964a2e09da499d3007a936e984d6" Nov 03 00:40:25.684680 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:40:25.684659 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d09bf8826c372ae6570466a0bd7132bced9964a2e09da499d3007a936e984d6\": container with ID starting with 5d09bf8826c372ae6570466a0bd7132bced9964a2e09da499d3007a936e984d6 not found: ID does not exist" containerID="5d09bf8826c372ae6570466a0bd7132bced9964a2e09da499d3007a936e984d6" Nov 03 00:40:25.684950 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.684686 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d09bf8826c372ae6570466a0bd7132bced9964a2e09da499d3007a936e984d6"} err="failed to get container status \"5d09bf8826c372ae6570466a0bd7132bced9964a2e09da499d3007a936e984d6\": rpc error: code = NotFound desc = could not find container \"5d09bf8826c372ae6570466a0bd7132bced9964a2e09da499d3007a936e984d6\": container with ID starting with 5d09bf8826c372ae6570466a0bd7132bced9964a2e09da499d3007a936e984d6 not found: ID does not exist" Nov 03 00:40:25.684950 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.684722 6400 scope.go:122] "RemoveContainer" containerID="aa42b5d0eb09dc1a0e6155b9ff15a59b978afdd8297081b8665cd0ee3cfce650" Nov 03 00:40:25.685123 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:40:25.685095 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa42b5d0eb09dc1a0e6155b9ff15a59b978afdd8297081b8665cd0ee3cfce650\": container with ID starting with aa42b5d0eb09dc1a0e6155b9ff15a59b978afdd8297081b8665cd0ee3cfce650 not found: ID does not exist" containerID="aa42b5d0eb09dc1a0e6155b9ff15a59b978afdd8297081b8665cd0ee3cfce650" Nov 03 00:40:25.685175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.685130 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa42b5d0eb09dc1a0e6155b9ff15a59b978afdd8297081b8665cd0ee3cfce650"} err="failed to get container status \"aa42b5d0eb09dc1a0e6155b9ff15a59b978afdd8297081b8665cd0ee3cfce650\": rpc error: code = NotFound desc = could not find container \"aa42b5d0eb09dc1a0e6155b9ff15a59b978afdd8297081b8665cd0ee3cfce650\": container with ID starting with aa42b5d0eb09dc1a0e6155b9ff15a59b978afdd8297081b8665cd0ee3cfce650 not found: ID does not exist" Nov 03 00:40:25.782588 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.782563 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd"] Nov 03 00:40:25.787454 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:25.787424 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-cb486a-openshift-default-58cd7b9cc-plcwd"] Nov 03 00:40:26.634194 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:26.634151 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts"] Nov 03 00:40:26.735509 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:26.735475 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:27.075285 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:27.075257 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:27.152418 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:40:27.152382 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:40:27.178302 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:27.178273 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" path="/var/lib/kubelet/pods/0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53/volumes" Nov 03 00:40:28.266178 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:28.266143 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:28.477643 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:28.477612 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:29.221350 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:29.221316 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:32.534855 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:32.534756 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:34.308899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:34.308868 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:36.710413 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:36.710377 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:37.875366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:37.875335 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:38.594001 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.593965 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq"] Nov 03 00:40:38.594672 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.594652 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerName="istio-proxy" Nov 03 00:40:38.594722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.594678 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerName="istio-proxy" Nov 03 00:40:38.594722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.594696 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerName="istio-proxy" Nov 03 00:40:38.594722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.594702 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerName="istio-proxy" Nov 03 00:40:38.594840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.594749 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e8e9bfae-6228-44d2-b1ec-e64ffbf6b105" containerName="maas-api" Nov 03 00:40:38.594840 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.594757 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e9bfae-6228-44d2-b1ec-e64ffbf6b105" containerName="maas-api" Nov 03 00:40:38.594917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.594897 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerName="istio-proxy" Nov 03 00:40:38.594917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.594914 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e9bfae-6228-44d2-b1ec-e64ffbf6b105" containerName="maas-api" Nov 03 00:40:38.594977 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.594926 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3c4cf1-3f8e-40a6-9fdf-16ad193f1a53" containerName="istio-proxy" Nov 03 00:40:38.600802 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.600778 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.665340 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.665310 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-envoy\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.665516 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.665348 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-credential-socket\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.665516 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.665434 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c3711f-87be-461b-a5d6-f7d6295b4002-istiod-ca-cert\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.665516 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.665492 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-certs\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.665636 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.665587 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-podinfo\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.665636 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.665616 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pw7g\" (UniqueName: \"kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-kube-api-access-6pw7g\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.665727 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.665717 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-token\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.665781 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.665768 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-data\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.665860 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.665802 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-socket\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.699981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.699949 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq"] Nov 03 00:40:38.704210 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.704185 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-d5db94-openshift-default-dockercfg-zqndh\"" Nov 03 00:40:38.712510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.712480 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:38.769897 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.768644 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-token\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.769897 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.769131 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-data\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.769897 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.769444 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-socket\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.769897 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.769707 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-envoy\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.769897 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.769795 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-credential-socket\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.772853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.770416 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-credential-socket\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.776853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.776087 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-socket\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.776853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.776370 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-data\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.776853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.776493 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c3711f-87be-461b-a5d6-f7d6295b4002-istiod-ca-cert\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.776853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.776537 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-certs\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.776853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.776613 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-podinfo\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.776853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.776655 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pw7g\" (UniqueName: \"kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-kube-api-access-6pw7g\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.777251 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.777078 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-certs\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.796846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.781416 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c3711f-87be-461b-a5d6-f7d6295b4002-istiod-ca-cert\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.801671 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.801322 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-envoy\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.804418 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.804385 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-podinfo\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.826249 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.826223 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-token\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.832904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.832876 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pw7g\" (UniqueName: \"kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-kube-api-access-6pw7g\") pod \"e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.919181 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.918704 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:38.942459 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:38.942368 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:39.310868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:39.310814 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:39.358338 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:39.353459 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq"] Nov 03 00:40:39.369535 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:39.369496 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:40:39.369659 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:39.369585 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:40:39.369659 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:39.369631 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:40:39.771914 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:39.771807 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" event={"ID":"15c3711f-87be-461b-a5d6-f7d6295b4002","Type":"ContainerStarted","Data":"3b65431c89dbdc12f7f50dba7b1abac2fbe9ad797cc5543afc0d87865955fdc9"} Nov 03 00:40:39.771914 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:39.771888 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" event={"ID":"15c3711f-87be-461b-a5d6-f7d6295b4002","Type":"ContainerStarted","Data":"daa4b1b01dcbf6e404eb7c0a7add06f2fb893f3b75b5a414950b11003f34df8a"} Nov 03 00:40:39.890540 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:39.890474 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" podStartSLOduration=1.890446867 podStartE2EDuration="1.890446867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:39.887332047 +0000 UTC m=+2086.838272205" watchObservedRunningTime="2026-11-03 00:40:39.890446867 +0000 UTC m=+2086.841387024" Nov 03 00:40:39.919018 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:39.918987 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:40.924545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:40.924511 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:41.658077 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:41.658028 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" podUID="af260334-9c1b-43e8-a923-ed50a40ecce8" containerName="main" containerID="cri-o://7615964ee0bd0a744b970f36803e7e2bc270fc9e6e050906bf5c97feca609331" gracePeriod=45 Nov 03 00:40:41.786577 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:41.786545 6400 generic.go:356] "Generic (PLEG): container finished" podID="af260334-9c1b-43e8-a923-ed50a40ecce8" containerID="7615964ee0bd0a744b970f36803e7e2bc270fc9e6e050906bf5c97feca609331" exitCode=0 Nov 03 00:40:41.786701 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:41.786607 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" event={"ID":"af260334-9c1b-43e8-a923-ed50a40ecce8","Type":"ContainerDied","Data":"7615964ee0bd0a744b970f36803e7e2bc270fc9e6e050906bf5c97feca609331"} Nov 03 00:40:41.786897 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:41.786879 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:41.787936 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:41.787916 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:40:41.909838 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:41.909778 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:40:42.022025 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.021996 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-tmp-dir\") pod \"af260334-9c1b-43e8-a923-ed50a40ecce8\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " Nov 03 00:40:42.022471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.022082 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-home\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-home\") pod \"af260334-9c1b-43e8-a923-ed50a40ecce8\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " Nov 03 00:40:42.022471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.022116 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-dshm\") pod \"af260334-9c1b-43e8-a923-ed50a40ecce8\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " Nov 03 00:40:42.022471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.022198 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af260334-9c1b-43e8-a923-ed50a40ecce8-kube-api-access-5wljf\" (UniqueName: \"kubernetes.io/projected/af260334-9c1b-43e8-a923-ed50a40ecce8-kube-api-access-5wljf\") pod \"af260334-9c1b-43e8-a923-ed50a40ecce8\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " Nov 03 00:40:42.022471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.022224 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af260334-9c1b-43e8-a923-ed50a40ecce8-tls-certs\" (UniqueName: \"kubernetes.io/secret/af260334-9c1b-43e8-a923-ed50a40ecce8-tls-certs\") pod \"af260334-9c1b-43e8-a923-ed50a40ecce8\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " Nov 03 00:40:42.022471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.022247 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-tmp-dir" pod "af260334-9c1b-43e8-a923-ed50a40ecce8" (UID: "af260334-9c1b-43e8-a923-ed50a40ecce8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:40:42.022471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.022260 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-model-cache\") pod \"af260334-9c1b-43e8-a923-ed50a40ecce8\" (UID: \"af260334-9c1b-43e8-a923-ed50a40ecce8\") " Nov 03 00:40:42.022471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.022379 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-home" pod "af260334-9c1b-43e8-a923-ed50a40ecce8" (UID: "af260334-9c1b-43e8-a923-ed50a40ecce8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:40:42.022768 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.022474 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-model-cache" pod "af260334-9c1b-43e8-a923-ed50a40ecce8" (UID: "af260334-9c1b-43e8-a923-ed50a40ecce8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:40:42.022768 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.022739 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:42.022768 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.022756 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:42.022768 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.022766 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:42.024595 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.024572 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af260334-9c1b-43e8-a923-ed50a40ecce8-kube-api-access-5wljf" pod "af260334-9c1b-43e8-a923-ed50a40ecce8" (UID: "af260334-9c1b-43e8-a923-ed50a40ecce8"). InnerVolumeSpecName "kube-api-access-5wljf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:40:42.025023 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.024992 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-dshm" pod "af260334-9c1b-43e8-a923-ed50a40ecce8" (UID: "af260334-9c1b-43e8-a923-ed50a40ecce8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:40:42.025088 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.025022 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af260334-9c1b-43e8-a923-ed50a40ecce8-tls-certs" pod "af260334-9c1b-43e8-a923-ed50a40ecce8" (UID: "af260334-9c1b-43e8-a923-ed50a40ecce8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:40:42.123975 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.123946 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wljf\" (UniqueName: \"kubernetes.io/projected/af260334-9c1b-43e8-a923-ed50a40ecce8-kube-api-access-5wljf\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:42.123975 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.123973 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af260334-9c1b-43e8-a923-ed50a40ecce8-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:42.123975 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.123986 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af260334-9c1b-43e8-a923-ed50a40ecce8-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:42.719701 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.719669 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:42.793893 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.793862 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" event={"ID":"af260334-9c1b-43e8-a923-ed50a40ecce8","Type":"ContainerDied","Data":"469062fab935e8aa83ec85f891cde3a11f4fd73255cf9ecc2da2ab81bd253c6e"} Nov 03 00:40:42.794066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.793897 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts" Nov 03 00:40:42.794066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.793906 6400 scope.go:122] "RemoveContainer" containerID="7615964ee0bd0a744b970f36803e7e2bc270fc9e6e050906bf5c97feca609331" Nov 03 00:40:42.948226 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.948191 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:42.959785 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:42.959749 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts"] Nov 03 00:40:43.009081 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:43.009027 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-1a883c/ipp-route-1a883c-kserve-f84f4484f-lk8ts"] Nov 03 00:40:43.177927 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:43.177902 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af260334-9c1b-43e8-a923-ed50a40ecce8" path="/var/lib/kubelet/pods/af260334-9c1b-43e8-a923-ed50a40ecce8/volumes" Nov 03 00:40:44.201844 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:44.201789 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:45.926176 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:45.926143 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:46.099626 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.099588 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:46.452563 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.452533 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9"] Nov 03 00:40:46.453108 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.453089 6400 cpu_manager.go:414] "containerMap: removing container" podUID="af260334-9c1b-43e8-a923-ed50a40ecce8" containerName="main" Nov 03 00:40:46.453108 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.453109 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af260334-9c1b-43e8-a923-ed50a40ecce8" containerName="main" Nov 03 00:40:46.453295 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.453213 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="af260334-9c1b-43e8-a923-ed50a40ecce8" containerName="main" Nov 03 00:40:46.456846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.456810 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" Nov 03 00:40:46.519846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.514237 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d5db94\"" Nov 03 00:40:46.564621 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.564594 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d5db94-dockercfg-q6zkb\"" Nov 03 00:40:46.570836 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.570783 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69c5308-3b24-401a-aab8-d3c60d85b690-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9\" (UID: \"d69c5308-3b24-401a-aab8-d3c60d85b690\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" Nov 03 00:40:46.571017 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.570858 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4mtk\" (UniqueName: \"kubernetes.io/projected/d69c5308-3b24-401a-aab8-d3c60d85b690-kube-api-access-v4mtk\") pod \"payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9\" (UID: \"d69c5308-3b24-401a-aab8-d3c60d85b690\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" Nov 03 00:40:46.571017 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.570935 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9"] Nov 03 00:40:46.657458 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.657415 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x"] Nov 03 00:40:46.661212 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.661191 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" Nov 03 00:40:46.673280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.673257 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69c5308-3b24-401a-aab8-d3c60d85b690-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9\" (UID: \"d69c5308-3b24-401a-aab8-d3c60d85b690\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" Nov 03 00:40:46.673389 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.673292 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4mtk\" (UniqueName: \"kubernetes.io/projected/d69c5308-3b24-401a-aab8-d3c60d85b690-kube-api-access-v4mtk\") pod \"payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9\" (UID: \"d69c5308-3b24-401a-aab8-d3c60d85b690\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" Nov 03 00:40:46.673811 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.673795 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69c5308-3b24-401a-aab8-d3c60d85b690-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9\" (UID: \"d69c5308-3b24-401a-aab8-d3c60d85b690\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" Nov 03 00:40:46.718592 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.718538 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x"] Nov 03 00:40:46.729560 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.729541 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4mtk\" (UniqueName: \"kubernetes.io/projected/d69c5308-3b24-401a-aab8-d3c60d85b690-kube-api-access-v4mtk\") pod \"payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9\" (UID: \"d69c5308-3b24-401a-aab8-d3c60d85b690\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" Nov 03 00:40:46.770771 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.770750 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" Nov 03 00:40:46.773933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.773838 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x\" (UID: \"d67b18c6-40ba-46e4-9e34-093a4dd6ec77\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" Nov 03 00:40:46.773933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.773904 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtw4v\" (UniqueName: \"kubernetes.io/projected/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-kube-api-access-vtw4v\") pod \"payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x\" (UID: \"d67b18c6-40ba-46e4-9e34-093a4dd6ec77\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" Nov 03 00:40:46.875006 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.874973 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x\" (UID: \"d67b18c6-40ba-46e4-9e34-093a4dd6ec77\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" Nov 03 00:40:46.875141 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.875024 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtw4v\" (UniqueName: \"kubernetes.io/projected/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-kube-api-access-vtw4v\") pod \"payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x\" (UID: \"d67b18c6-40ba-46e4-9e34-093a4dd6ec77\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" Nov 03 00:40:46.875648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.875627 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x\" (UID: \"d67b18c6-40ba-46e4-9e34-093a4dd6ec77\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" Nov 03 00:40:46.931448 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.931423 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtw4v\" (UniqueName: \"kubernetes.io/projected/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-kube-api-access-vtw4v\") pod \"payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x\" (UID: \"d67b18c6-40ba-46e4-9e34-093a4dd6ec77\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" Nov 03 00:40:46.973532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:46.973478 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" Nov 03 00:40:47.145763 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:47.145738 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9"] Nov 03 00:40:47.147270 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:40:47.147243 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69c5308_3b24_401a_aab8_d3c60d85b690.slice/crio-fdf83e996cc1b781f0a1cdd74453fef8666b30e165fd2c17940ca912310fcfd7 WatchSource:0}: Error finding container fdf83e996cc1b781f0a1cdd74453fef8666b30e165fd2c17940ca912310fcfd7: Status 404 returned error can't find the container with id fdf83e996cc1b781f0a1cdd74453fef8666b30e165fd2c17940ca912310fcfd7 Nov 03 00:40:47.344923 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:47.344900 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x"] Nov 03 00:40:47.346625 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:40:47.346606 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67b18c6_40ba_46e4_9e34_093a4dd6ec77.slice/crio-28a19e3add23856eaa2a6c8c358668e21d67cecdadca1aa20ae2fe86635bea23 WatchSource:0}: Error finding container 28a19e3add23856eaa2a6c8c358668e21d67cecdadca1aa20ae2fe86635bea23: Status 404 returned error can't find the container with id 28a19e3add23856eaa2a6c8c358668e21d67cecdadca1aa20ae2fe86635bea23 Nov 03 00:40:47.823382 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:47.823346 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" event={"ID":"d69c5308-3b24-401a-aab8-d3c60d85b690","Type":"ContainerStarted","Data":"a67428e7c59ec0847aaf9263cd4a2229b3242affb25c152cd7624f6985785ad2"} Nov 03 00:40:47.823587 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:47.823417 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" event={"ID":"d69c5308-3b24-401a-aab8-d3c60d85b690","Type":"ContainerStarted","Data":"fdf83e996cc1b781f0a1cdd74453fef8666b30e165fd2c17940ca912310fcfd7"} Nov 03 00:40:47.823587 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:47.823470 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" Nov 03 00:40:47.825006 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:47.824981 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" event={"ID":"d67b18c6-40ba-46e4-9e34-093a4dd6ec77","Type":"ContainerStarted","Data":"819563eb8f88a2d1a5690f8914c0e7ce6bd7ae07fa3b81cb89e9a5e498b8cc10"} Nov 03 00:40:47.825120 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:47.825022 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" event={"ID":"d67b18c6-40ba-46e4-9e34-093a4dd6ec77","Type":"ContainerStarted","Data":"28a19e3add23856eaa2a6c8c358668e21d67cecdadca1aa20ae2fe86635bea23"} Nov 03 00:40:47.825120 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:47.825086 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" Nov 03 00:40:47.938737 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:47.938689 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" podStartSLOduration=1.938673696 podStartE2EDuration="1.938673696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:47.935518188 +0000 UTC m=+2094.886458341" watchObservedRunningTime="2026-11-03 00:40:47.938673696 +0000 UTC m=+2094.889613853" Nov 03 00:40:49.411877 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:49.411845 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:52.239805 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:52.239763 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:53.469466 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:53.469418 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" podStartSLOduration=7.469405454 podStartE2EDuration="7.469405454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:48.064850766 +0000 UTC m=+2095.015790924" watchObservedRunningTime="2026-11-03 00:40:53.469405454 +0000 UTC m=+2100.420345610" Nov 03 00:40:53.471745 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:53.471720 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:40:54.345128 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:54.345092 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:57.151777 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:40:57.151741 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:40:58.832631 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:58.832603 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" Nov 03 00:40:58.833067 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:58.832880 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" Nov 03 00:40:59.213722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:40:59.213641 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:02.774016 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:02.773975 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:03.910154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:03.910113 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:04.182966 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:04.182890 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:04.350005 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:04.349974 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:09.055514 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:09.055467 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:09.594136 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:09.594108 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:10.381677 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:10.381645 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:13.351452 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.351423 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt"] Nov 03 00:41:13.356149 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.356131 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:13.411055 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.411024 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-bdd194-serving-cert\"" Nov 03 00:41:13.462687 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.462648 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt"] Nov 03 00:41:13.500513 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.500480 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-metrics-certs\") pod \"maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:13.500684 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.500616 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:13.500684 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.500661 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cf9519-081f-4426-a37a-789cb95c4a4a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:13.500932 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.500911 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht5cd\" (UniqueName: \"kubernetes.io/projected/43cf9519-081f-4426-a37a-789cb95c4a4a-kube-api-access-ht5cd\") pod \"maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:13.569462 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.569428 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:13.616620 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.616528 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-metrics-certs\") pod \"maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:13.617406 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.617378 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:13.617702 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.617683 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cf9519-081f-4426-a37a-789cb95c4a4a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:13.617870 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:41:13.617641 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-serving-cert: secret "maas-api-e2e-praxis-b-bdd194-serving-cert" not found Nov 03 00:41:13.621660 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:41:13.621637 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-maas-api-tls podName:43cf9519-081f-4426-a37a-789cb95c4a4a nodeName:}" failed. No retries permitted until 2026-11-03 00:41:14.12160918 +0000 UTC m=+2121.072549326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-maas-api-tls") pod "maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" (UID: "43cf9519-081f-4426-a37a-789cb95c4a4a") : secret "maas-api-e2e-praxis-b-bdd194-serving-cert" not found Nov 03 00:41:13.621848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.618107 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht5cd\" (UniqueName: \"kubernetes.io/projected/43cf9519-081f-4426-a37a-789cb95c4a4a-kube-api-access-ht5cd\") pod \"maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:13.622792 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.622677 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cf9519-081f-4426-a37a-789cb95c4a4a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:13.644387 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.644356 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-metrics-certs\") pod \"maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:13.674801 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.674773 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht5cd\" (UniqueName: \"kubernetes.io/projected/43cf9519-081f-4426-a37a-789cb95c4a4a-kube-api-access-ht5cd\") pod \"maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:13.753336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.753298 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:13.905802 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:13.905711 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:14.133992 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:14.133962 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:14.136421 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:14.136398 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:14.269660 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:14.269590 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:14.645133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:14.645109 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt"] Nov 03 00:41:14.647406 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:41:14.647369 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cf9519_081f_4426_a37a_789cb95c4a4a.slice/crio-ff34f538900316d060b5659906b912d43a71c26a954ed720fd6e2d612727b384 WatchSource:0}: Error finding container ff34f538900316d060b5659906b912d43a71c26a954ed720fd6e2d612727b384: Status 404 returned error can't find the container with id ff34f538900316d060b5659906b912d43a71c26a954ed720fd6e2d612727b384 Nov 03 00:41:14.897957 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:14.897867 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:14.990044 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:14.990013 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" event={"ID":"43cf9519-081f-4426-a37a-789cb95c4a4a","Type":"ContainerStarted","Data":"ff34f538900316d060b5659906b912d43a71c26a954ed720fd6e2d612727b384"} Nov 03 00:41:15.996836 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:15.996787 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" event={"ID":"43cf9519-081f-4426-a37a-789cb95c4a4a","Type":"ContainerStarted","Data":"d74f17e703c6ff28cbb5edff03664df641036431cf74f9e30eaf3f76391eeede"} Nov 03 00:41:15.997312 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:15.996974 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:16.051235 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:16.051201 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:16.112245 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:16.112193 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" podStartSLOduration=2.2645573 podStartE2EDuration="3.112178305s" totalImagesPullingTime="847.621005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:16.110726183 +0000 UTC m=+2123.061666339" watchObservedRunningTime="2026-11-03 00:41:16.112178305 +0000 UTC m=+2123.063118460" Nov 03 00:41:17.952750 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:17.952718 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:18.160907 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.160873 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9"] Nov 03 00:41:18.161115 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.161095 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" podUID="98dc7535-a699-4e56-bc44-b44bfa015d6f" containerName="maas-api" containerID="cri-o://8ae3ad2ff7686d8f4048b7b7a6ce888085b1706f182c363b2946d394550b76dc" gracePeriod=30 Nov 03 00:41:18.323989 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.323954 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:18.421485 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.421455 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:41:18.587395 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.587363 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-metrics-certs\") pod \"98dc7535-a699-4e56-bc44-b44bfa015d6f\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " Nov 03 00:41:18.587576 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.587477 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98dc7535-a699-4e56-bc44-b44bfa015d6f-kube-api-access-zk9fh\" (UniqueName: \"kubernetes.io/projected/98dc7535-a699-4e56-bc44-b44bfa015d6f-kube-api-access-zk9fh\") pod \"98dc7535-a699-4e56-bc44-b44bfa015d6f\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " Nov 03 00:41:18.587644 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.587573 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-maas-api-tls\") pod \"98dc7535-a699-4e56-bc44-b44bfa015d6f\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " Nov 03 00:41:18.587644 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.587603 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98dc7535-a699-4e56-bc44-b44bfa015d6f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dc7535-a699-4e56-bc44-b44bfa015d6f-openshift-service-ca\") pod \"98dc7535-a699-4e56-bc44-b44bfa015d6f\" (UID: \"98dc7535-a699-4e56-bc44-b44bfa015d6f\") " Nov 03 00:41:18.588154 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.588109 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98dc7535-a699-4e56-bc44-b44bfa015d6f-openshift-service-ca" pod "98dc7535-a699-4e56-bc44-b44bfa015d6f" (UID: "98dc7535-a699-4e56-bc44-b44bfa015d6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:18.589895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.589869 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-metrics-certs" pod "98dc7535-a699-4e56-bc44-b44bfa015d6f" (UID: "98dc7535-a699-4e56-bc44-b44bfa015d6f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:18.590210 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.590192 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-maas-api-tls" pod "98dc7535-a699-4e56-bc44-b44bfa015d6f" (UID: "98dc7535-a699-4e56-bc44-b44bfa015d6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:18.590334 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.590311 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98dc7535-a699-4e56-bc44-b44bfa015d6f-kube-api-access-zk9fh" pod "98dc7535-a699-4e56-bc44-b44bfa015d6f" (UID: "98dc7535-a699-4e56-bc44-b44bfa015d6f"). InnerVolumeSpecName "kube-api-access-zk9fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:18.689733 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.689685 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:18.689733 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.689719 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dc7535-a699-4e56-bc44-b44bfa015d6f-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:18.689733 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.689729 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dc7535-a699-4e56-bc44-b44bfa015d6f-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:18.689733 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.689739 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk9fh\" (UniqueName: \"kubernetes.io/projected/98dc7535-a699-4e56-bc44-b44bfa015d6f-kube-api-access-zk9fh\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:18.876546 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:18.876461 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:19.017542 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:19.017510 6400 generic.go:356] "Generic (PLEG): container finished" podID="98dc7535-a699-4e56-bc44-b44bfa015d6f" containerID="8ae3ad2ff7686d8f4048b7b7a6ce888085b1706f182c363b2946d394550b76dc" exitCode=0 Nov 03 00:41:19.018004 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:19.017563 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" Nov 03 00:41:19.018004 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:19.017575 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" event={"ID":"98dc7535-a699-4e56-bc44-b44bfa015d6f","Type":"ContainerDied","Data":"8ae3ad2ff7686d8f4048b7b7a6ce888085b1706f182c363b2946d394550b76dc"} Nov 03 00:41:19.018004 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:19.017614 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9" event={"ID":"98dc7535-a699-4e56-bc44-b44bfa015d6f","Type":"ContainerDied","Data":"4948c531eb8a96e58ed9708612bc86569b9cb7d8c7deea97e36d4de144ef2123"} Nov 03 00:41:19.018004 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:19.017635 6400 scope.go:122] "RemoveContainer" containerID="8ae3ad2ff7686d8f4048b7b7a6ce888085b1706f182c363b2946d394550b76dc" Nov 03 00:41:19.028000 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:19.027981 6400 scope.go:122] "RemoveContainer" containerID="8ae3ad2ff7686d8f4048b7b7a6ce888085b1706f182c363b2946d394550b76dc" Nov 03 00:41:19.028272 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:41:19.028253 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae3ad2ff7686d8f4048b7b7a6ce888085b1706f182c363b2946d394550b76dc\": container with ID starting with 8ae3ad2ff7686d8f4048b7b7a6ce888085b1706f182c363b2946d394550b76dc not found: ID does not exist" containerID="8ae3ad2ff7686d8f4048b7b7a6ce888085b1706f182c363b2946d394550b76dc" Nov 03 00:41:19.028312 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:19.028285 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae3ad2ff7686d8f4048b7b7a6ce888085b1706f182c363b2946d394550b76dc"} err="failed to get container status \"8ae3ad2ff7686d8f4048b7b7a6ce888085b1706f182c363b2946d394550b76dc\": rpc error: code = NotFound desc = could not find container \"8ae3ad2ff7686d8f4048b7b7a6ce888085b1706f182c363b2946d394550b76dc\": container with ID starting with 8ae3ad2ff7686d8f4048b7b7a6ce888085b1706f182c363b2946d394550b76dc not found: ID does not exist" Nov 03 00:41:19.096365 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:19.096335 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:19.182719 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:19.182648 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9"] Nov 03 00:41:19.187073 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:19.187045 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1a883c-74d98fb786-9d5f9"] Nov 03 00:41:20.276019 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:20.275979 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:21.178871 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:21.178841 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98dc7535-a699-4e56-bc44-b44bfa015d6f" path="/var/lib/kubelet/pods/98dc7535-a699-4e56-bc44-b44bfa015d6f/volumes" Nov 03 00:41:21.342095 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:21.342057 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:22.010569 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:22.010541 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:41:22.046426 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:22.046398 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:22.247035 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:22.247003 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:23.573123 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:23.573091 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:24.065273 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:24.065233 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:24.516737 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:24.516652 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:24.824173 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:24.824140 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:25.030903 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:25.030872 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:25.219040 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:25.218952 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:25.585649 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:25.585616 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:27.151556 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:41:27.151521 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:41:30.251569 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:30.251539 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:31.200217 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.200177 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:31.556691 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.556659 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq"] Nov 03 00:41:31.557179 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.556922 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" podUID="b16a4886-e342-4469-a1eb-77a97f8ad0d9" containerName="maas-api" containerID="cri-o://d3723b50a3500123ed77fa77e27bb2984550d4efc146d34a689ab821ae3507af" gracePeriod=30 Nov 03 00:41:31.676695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.676627 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:31.895626 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.895598 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:41:31.914791 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.914767 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-maas-api-tls\") pod \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " Nov 03 00:41:31.914930 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.914910 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16a4886-e342-4469-a1eb-77a97f8ad0d9-kube-api-access-5xg2g\" (UniqueName: \"kubernetes.io/projected/b16a4886-e342-4469-a1eb-77a97f8ad0d9-kube-api-access-5xg2g\") pod \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " Nov 03 00:41:31.914994 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.914980 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16a4886-e342-4469-a1eb-77a97f8ad0d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16a4886-e342-4469-a1eb-77a97f8ad0d9-openshift-service-ca\") pod \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " Nov 03 00:41:31.917013 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.915116 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-metrics-certs\") pod \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\" (UID: \"b16a4886-e342-4469-a1eb-77a97f8ad0d9\") " Nov 03 00:41:31.917013 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.915945 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16a4886-e342-4469-a1eb-77a97f8ad0d9-openshift-service-ca" pod "b16a4886-e342-4469-a1eb-77a97f8ad0d9" (UID: "b16a4886-e342-4469-a1eb-77a97f8ad0d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:31.923961 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.923935 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-maas-api-tls" pod "b16a4886-e342-4469-a1eb-77a97f8ad0d9" (UID: "b16a4886-e342-4469-a1eb-77a97f8ad0d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:31.926051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.924658 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:31.926051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.924692 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16a4886-e342-4469-a1eb-77a97f8ad0d9-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:31.944851 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.941318 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16a4886-e342-4469-a1eb-77a97f8ad0d9-kube-api-access-5xg2g" pod "b16a4886-e342-4469-a1eb-77a97f8ad0d9" (UID: "b16a4886-e342-4469-a1eb-77a97f8ad0d9"). InnerVolumeSpecName "kube-api-access-5xg2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:31.966845 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:31.962980 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-metrics-certs" pod "b16a4886-e342-4469-a1eb-77a97f8ad0d9" (UID: "b16a4886-e342-4469-a1eb-77a97f8ad0d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:32.026119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:32.026087 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xg2g\" (UniqueName: \"kubernetes.io/projected/b16a4886-e342-4469-a1eb-77a97f8ad0d9-kube-api-access-5xg2g\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:32.026119 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:32.026117 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b16a4886-e342-4469-a1eb-77a97f8ad0d9-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:32.156892 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:32.153203 6400 generic.go:356] "Generic (PLEG): container finished" podID="b16a4886-e342-4469-a1eb-77a97f8ad0d9" containerID="d3723b50a3500123ed77fa77e27bb2984550d4efc146d34a689ab821ae3507af" exitCode=0 Nov 03 00:41:32.156892 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:32.153386 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" event={"ID":"b16a4886-e342-4469-a1eb-77a97f8ad0d9","Type":"ContainerDied","Data":"d3723b50a3500123ed77fa77e27bb2984550d4efc146d34a689ab821ae3507af"} Nov 03 00:41:32.156892 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:32.153477 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" event={"ID":"b16a4886-e342-4469-a1eb-77a97f8ad0d9","Type":"ContainerDied","Data":"675b10ff01e9883629ab255bcba39cf4e544f05ace7dbffe15d29c9c4bef61ea"} Nov 03 00:41:32.156892 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:32.153521 6400 scope.go:122] "RemoveContainer" containerID="d3723b50a3500123ed77fa77e27bb2984550d4efc146d34a689ab821ae3507af" Nov 03 00:41:32.156892 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:32.153715 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" Nov 03 00:41:32.191680 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:32.191659 6400 scope.go:122] "RemoveContainer" containerID="d3723b50a3500123ed77fa77e27bb2984550d4efc146d34a689ab821ae3507af" Nov 03 00:41:32.192003 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:41:32.191983 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3723b50a3500123ed77fa77e27bb2984550d4efc146d34a689ab821ae3507af\": container with ID starting with d3723b50a3500123ed77fa77e27bb2984550d4efc146d34a689ab821ae3507af not found: ID does not exist" containerID="d3723b50a3500123ed77fa77e27bb2984550d4efc146d34a689ab821ae3507af" Nov 03 00:41:32.192065 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:32.192013 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3723b50a3500123ed77fa77e27bb2984550d4efc146d34a689ab821ae3507af"} err="failed to get container status \"d3723b50a3500123ed77fa77e27bb2984550d4efc146d34a689ab821ae3507af\": rpc error: code = NotFound desc = could not find container \"d3723b50a3500123ed77fa77e27bb2984550d4efc146d34a689ab821ae3507af\": container with ID starting with d3723b50a3500123ed77fa77e27bb2984550d4efc146d34a689ab821ae3507af not found: ID does not exist" Nov 03 00:41:32.318624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:32.318595 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq"] Nov 03 00:41:32.321629 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:32.321602 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq"] Nov 03 00:41:33.178579 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:33.178549 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b16a4886-e342-4469-a1eb-77a97f8ad0d9" path="/var/lib/kubelet/pods/b16a4886-e342-4469-a1eb-77a97f8ad0d9/volumes" Nov 03 00:41:33.601424 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:33.601388 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:34.817437 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:34.817395 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0db2fb-76688d856c-nvwbq" podUID="b16a4886-e342-4469-a1eb-77a97f8ad0d9" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.90:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:41:38.345596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:38.345558 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:41:38.853480 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:38.853442 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:40.065718 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:40.065679 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:41.007152 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:41.007117 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:41.215394 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:41.215368 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:42.661599 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:42.661565 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:43.101471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:43.101436 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:44.014237 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:44.014203 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:48.730501 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.730460 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n"] Nov 03 00:41:48.731066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.731046 6400 cpu_manager.go:414] "containerMap: removing container" podUID="b16a4886-e342-4469-a1eb-77a97f8ad0d9" containerName="maas-api" Nov 03 00:41:48.731121 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.731068 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16a4886-e342-4469-a1eb-77a97f8ad0d9" containerName="maas-api" Nov 03 00:41:48.731121 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.731101 6400 cpu_manager.go:414] "containerMap: removing container" podUID="98dc7535-a699-4e56-bc44-b44bfa015d6f" containerName="maas-api" Nov 03 00:41:48.731121 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.731106 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98dc7535-a699-4e56-bc44-b44bfa015d6f" containerName="maas-api" Nov 03 00:41:48.731222 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.731181 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="98dc7535-a699-4e56-bc44-b44bfa015d6f" containerName="maas-api" Nov 03 00:41:48.731222 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.731190 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16a4886-e342-4469-a1eb-77a97f8ad0d9" containerName="maas-api" Nov 03 00:41:48.734609 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.734591 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:48.786660 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.786616 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-d5db94-serving-cert\"" Nov 03 00:41:48.831698 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.831661 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b438591-e7cd-4569-af4f-95254774c42e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:48.831911 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.831802 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-metrics-certs\") pod \"maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:48.831972 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.831955 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:48.832075 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.832055 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plvft\" (UniqueName: \"kubernetes.io/projected/2b438591-e7cd-4569-af4f-95254774c42e-kube-api-access-plvft\") pod \"maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:48.838147 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.838124 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n"] Nov 03 00:41:48.933434 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.933389 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvft\" (UniqueName: \"kubernetes.io/projected/2b438591-e7cd-4569-af4f-95254774c42e-kube-api-access-plvft\") pod \"maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:48.933618 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.933456 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b438591-e7cd-4569-af4f-95254774c42e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:48.933618 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.933507 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-metrics-certs\") pod \"maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:48.933618 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.933560 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:48.933739 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:41:48.933671 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-serving-cert: secret "maas-api-e2e-praxis-b-d5db94-serving-cert" not found Nov 03 00:41:48.933787 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:41:48.933745 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-maas-api-tls podName:2b438591-e7cd-4569-af4f-95254774c42e nodeName:}" failed. No retries permitted until 2026-11-03 00:41:49.433718659 +0000 UTC m=+2156.384658807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-maas-api-tls") pod "maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" (UID: "2b438591-e7cd-4569-af4f-95254774c42e") : secret "maas-api-e2e-praxis-b-d5db94-serving-cert" not found Nov 03 00:41:48.934180 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.934162 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b438591-e7cd-4569-af4f-95254774c42e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:48.936149 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.936130 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-metrics-certs\") pod \"maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:48.962450 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:48.962407 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:49.000454 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:49.000338 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvft\" (UniqueName: \"kubernetes.io/projected/2b438591-e7cd-4569-af4f-95254774c42e-kube-api-access-plvft\") pod \"maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:49.133414 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:49.133379 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:49.291544 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:49.291510 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:49.444723 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:49.444676 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:49.447342 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:49.447318 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:49.650238 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:49.650146 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:50.025464 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:50.025442 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n"] Nov 03 00:41:50.027544 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:41:50.027517 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b438591_e7cd_4569_af4f_95254774c42e.slice/crio-ccb08cb22e405384d90d72df3240bf1baf8f644c0b1deb456f69c7da19a3fcac WatchSource:0}: Error finding container ccb08cb22e405384d90d72df3240bf1baf8f644c0b1deb456f69c7da19a3fcac: Status 404 returned error can't find the container with id ccb08cb22e405384d90d72df3240bf1baf8f644c0b1deb456f69c7da19a3fcac Nov 03 00:41:50.290308 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:50.290273 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" event={"ID":"2b438591-e7cd-4569-af4f-95254774c42e","Type":"ContainerStarted","Data":"ccb08cb22e405384d90d72df3240bf1baf8f644c0b1deb456f69c7da19a3fcac"} Nov 03 00:41:51.298212 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:51.298172 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" event={"ID":"2b438591-e7cd-4569-af4f-95254774c42e","Type":"ContainerStarted","Data":"8be6a5a73936241a0c93f749a1c37254829c5ef5040033f0528b1b2e5c0f44ea"} Nov 03 00:41:51.298731 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:51.298323 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:51.412743 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:51.412688 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" podStartSLOduration=2.510928852 podStartE2EDuration="3.412671739s" totalImagesPullingTime="901.742887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:51.411204522 +0000 UTC m=+2158.362144692" watchObservedRunningTime="2026-11-03 00:41:51.412671739 +0000 UTC m=+2158.363611896" Nov 03 00:41:53.861560 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:53.861521 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:57.152563 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:41:57.152525 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:41:57.311271 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:57.311238 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:41:58.633256 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:41:58.633216 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:02.620613 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:02.620527 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj"] Nov 03 00:42:02.621024 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:02.620756 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" podUID="1c854675-3a17-4ead-ac15-3364f3de99f5" containerName="maas-api" containerID="cri-o://a7738e1a0f58290bcc98f8cb2c62b9a096dea5fa3a18ac4fa6e060f05ae32689" gracePeriod=30 Nov 03 00:42:02.748286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:02.748250 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:02.932045 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:02.932009 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:42:03.013959 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.013912 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-metrics-certs\") pod \"1c854675-3a17-4ead-ac15-3364f3de99f5\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " Nov 03 00:42:03.014174 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.014000 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c854675-3a17-4ead-ac15-3364f3de99f5-kube-api-access-4qxjk\" (UniqueName: \"kubernetes.io/projected/1c854675-3a17-4ead-ac15-3364f3de99f5-kube-api-access-4qxjk\") pod \"1c854675-3a17-4ead-ac15-3364f3de99f5\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " Nov 03 00:42:03.014174 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.014088 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c854675-3a17-4ead-ac15-3364f3de99f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c854675-3a17-4ead-ac15-3364f3de99f5-openshift-service-ca\") pod \"1c854675-3a17-4ead-ac15-3364f3de99f5\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " Nov 03 00:42:03.014174 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.014149 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-maas-api-tls\") pod \"1c854675-3a17-4ead-ac15-3364f3de99f5\" (UID: \"1c854675-3a17-4ead-ac15-3364f3de99f5\") " Nov 03 00:42:03.014501 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.014465 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c854675-3a17-4ead-ac15-3364f3de99f5-openshift-service-ca" pod "1c854675-3a17-4ead-ac15-3364f3de99f5" (UID: "1c854675-3a17-4ead-ac15-3364f3de99f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:03.014681 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.014663 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c854675-3a17-4ead-ac15-3364f3de99f5-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:03.016869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.016805 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-metrics-certs" pod "1c854675-3a17-4ead-ac15-3364f3de99f5" (UID: "1c854675-3a17-4ead-ac15-3364f3de99f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:03.027717 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.027675 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c854675-3a17-4ead-ac15-3364f3de99f5-kube-api-access-4qxjk" pod "1c854675-3a17-4ead-ac15-3364f3de99f5" (UID: "1c854675-3a17-4ead-ac15-3364f3de99f5"). InnerVolumeSpecName "kube-api-access-4qxjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:03.028078 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.028047 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-maas-api-tls" pod "1c854675-3a17-4ead-ac15-3364f3de99f5" (UID: "1c854675-3a17-4ead-ac15-3364f3de99f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:03.119205 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.119169 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:03.123507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.123443 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qxjk\" (UniqueName: \"kubernetes.io/projected/1c854675-3a17-4ead-ac15-3364f3de99f5-kube-api-access-4qxjk\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:03.123507 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.123479 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c854675-3a17-4ead-ac15-3364f3de99f5-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:03.365496 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.365457 6400 generic.go:356] "Generic (PLEG): container finished" podID="1c854675-3a17-4ead-ac15-3364f3de99f5" containerID="a7738e1a0f58290bcc98f8cb2c62b9a096dea5fa3a18ac4fa6e060f05ae32689" exitCode=0 Nov 03 00:42:03.365700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.365503 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" Nov 03 00:42:03.365700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.365540 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" event={"ID":"1c854675-3a17-4ead-ac15-3364f3de99f5","Type":"ContainerDied","Data":"a7738e1a0f58290bcc98f8cb2c62b9a096dea5fa3a18ac4fa6e060f05ae32689"} Nov 03 00:42:03.365700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.365605 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj" event={"ID":"1c854675-3a17-4ead-ac15-3364f3de99f5","Type":"ContainerDied","Data":"1b5519430de4c5bff68b42a67642212b189d4d5159e0205c336e592ef503cf87"} Nov 03 00:42:03.365700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.365634 6400 scope.go:122] "RemoveContainer" containerID="a7738e1a0f58290bcc98f8cb2c62b9a096dea5fa3a18ac4fa6e060f05ae32689" Nov 03 00:42:03.375770 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.375745 6400 scope.go:122] "RemoveContainer" containerID="a7738e1a0f58290bcc98f8cb2c62b9a096dea5fa3a18ac4fa6e060f05ae32689" Nov 03 00:42:03.376139 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:42:03.376110 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7738e1a0f58290bcc98f8cb2c62b9a096dea5fa3a18ac4fa6e060f05ae32689\": container with ID starting with a7738e1a0f58290bcc98f8cb2c62b9a096dea5fa3a18ac4fa6e060f05ae32689 not found: ID does not exist" containerID="a7738e1a0f58290bcc98f8cb2c62b9a096dea5fa3a18ac4fa6e060f05ae32689" Nov 03 00:42:03.376245 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.376146 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7738e1a0f58290bcc98f8cb2c62b9a096dea5fa3a18ac4fa6e060f05ae32689"} err="failed to get container status \"a7738e1a0f58290bcc98f8cb2c62b9a096dea5fa3a18ac4fa6e060f05ae32689\": rpc error: code = NotFound desc = could not find container \"a7738e1a0f58290bcc98f8cb2c62b9a096dea5fa3a18ac4fa6e060f05ae32689\": container with ID starting with a7738e1a0f58290bcc98f8cb2c62b9a096dea5fa3a18ac4fa6e060f05ae32689 not found: ID does not exist" Nov 03 00:42:03.531861 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.531359 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj"] Nov 03 00:42:03.539162 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:03.539125 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-75dfee-6fc6d958cb-4t2sj"] Nov 03 00:42:04.505055 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:04.505018 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:05.179531 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:05.179492 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c854675-3a17-4ead-ac15-3364f3de99f5" path="/var/lib/kubelet/pods/1c854675-3a17-4ead-ac15-3364f3de99f5/volumes" Nov 03 00:42:09.228067 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:09.227879 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:10.736576 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:10.736539 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:11.405323 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:11.405285 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:11.593964 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:11.593921 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:13.029838 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:13.029795 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:13.630218 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:13.630181 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:14.161354 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:14.161323 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:14.436722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:14.436631 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:16.425989 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:16.425953 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:42:19.223193 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:19.223146 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:20.257677 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:20.257644 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:20.441848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:20.441420 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:20.801846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:20.801473 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:24.405629 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:24.405587 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:26.460266 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:26.460176 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:26.500343 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:26.500314 6400 generic.go:356] "Generic (PLEG): container finished" podID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerID="3b65431c89dbdc12f7f50dba7b1abac2fbe9ad797cc5543afc0d87865955fdc9" exitCode=0 Nov 03 00:42:26.500511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:26.500372 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" event={"ID":"15c3711f-87be-461b-a5d6-f7d6295b4002","Type":"ContainerDied","Data":"3b65431c89dbdc12f7f50dba7b1abac2fbe9ad797cc5543afc0d87865955fdc9"} Nov 03 00:42:26.500811 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:26.500792 6400 scope.go:122] "RemoveContainer" containerID="3b65431c89dbdc12f7f50dba7b1abac2fbe9ad797cc5543afc0d87865955fdc9" Nov 03 00:42:26.502318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:26.502289 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:42:26.502392 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:26.502362 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:42:26.502427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:26.502396 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:42:26.786939 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:26.786912 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:42:27.153121 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:42:27.153014 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:42:27.508205 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:27.508123 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" event={"ID":"15c3711f-87be-461b-a5d6-f7d6295b4002","Type":"ContainerStarted","Data":"b69f258218aa582cdcc67b2db750e1bbe7d706e777c9aef62a2cd7d91c099bca"} Nov 03 00:42:27.919122 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:27.919089 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:42:27.920476 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:27.920455 6400 patch_prober.go:29] interesting pod/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:42:27.920530 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:27.920504 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" podUID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 00:42:28.041758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:28.041723 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:28.919919 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:28.919881 6400 patch_prober.go:29] interesting pod/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:42:28.919919 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:28.919934 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" podUID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 00:42:28.920790 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:28.920761 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:29.443394 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:29.443363 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:29.611362 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:29.611320 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:29.919995 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:29.919860 6400 patch_prober.go:29] interesting pod/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:42:29.919995 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:29.919931 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" podUID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 00:42:30.763431 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:30.763400 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:30.919394 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:30.919346 6400 patch_prober.go:29] interesting pod/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:42:30.919607 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:30.919409 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" podUID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 00:42:31.919491 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:31.919458 6400 patch_prober.go:29] interesting pod/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:42:31.919974 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:31.919513 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" podUID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 00:42:32.140293 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:32.140252 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:32.924037 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:32.924005 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:42:32.924454 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:32.924261 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:42:32.924992 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:32.924971 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:42:35.520855 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:35.520802 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:39.495085 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:39.495046 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:39.739860 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:39.739801 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:40.332326 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:40.332282 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:41.053011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:41.052975 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:41.558466 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:41.558431 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:43.160679 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:43.160642 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:42:45.604227 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:45.604193 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:47.207896 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.207856 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh"] Nov 03 00:42:47.208574 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.208556 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1c854675-3a17-4ead-ac15-3364f3de99f5" containerName="maas-api" Nov 03 00:42:47.208628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.208577 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c854675-3a17-4ead-ac15-3364f3de99f5" containerName="maas-api" Nov 03 00:42:47.208679 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.208665 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c854675-3a17-4ead-ac15-3364f3de99f5" containerName="maas-api" Nov 03 00:42:47.211971 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.211956 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.263645 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.263615 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-531ee4-openshift-default-dockercfg-pcjd8\"" Nov 03 00:42:47.315511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.315480 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh"] Nov 03 00:42:47.333299 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.333265 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbzd\" (UniqueName: \"kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-kube-api-access-tlbzd\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.333488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.333308 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6b2116-c767-4c94-867f-448307c7bb4f-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.333488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.333332 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.333488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.333394 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.333488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.333464 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.333629 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.333523 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.333629 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.333581 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.333629 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.333606 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.333767 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.333640 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.434968 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.434933 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.435187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.435009 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.435187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.435057 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.435187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.435094 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.435187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.435135 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.435409 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.435203 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbzd\" (UniqueName: \"kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-kube-api-access-tlbzd\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.435409 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.435249 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6b2116-c767-4c94-867f-448307c7bb4f-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.435409 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.435287 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.435409 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.435306 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.435900 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.435870 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.436054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.435935 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.436054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.435975 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.436054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.436004 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.436376 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.436354 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6b2116-c767-4c94-867f-448307c7bb4f-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.438035 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.438017 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.438138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.438123 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.491885 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.491794 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.492112 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.492086 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbzd\" (UniqueName: \"kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-kube-api-access-tlbzd\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.525846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.525785 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:47.734356 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.734322 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:47.950457 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.950425 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh"] Nov 03 00:42:47.952018 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:42:47.951990 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6b2116_c767_4c94_867f_448307c7bb4f.slice/crio-6276bbb512c7860176af6f14fa0bd18959a5ebcf3a33f90023ab96256ff14e62 WatchSource:0}: Error finding container 6276bbb512c7860176af6f14fa0bd18959a5ebcf3a33f90023ab96256ff14e62: Status 404 returned error can't find the container with id 6276bbb512c7860176af6f14fa0bd18959a5ebcf3a33f90023ab96256ff14e62 Nov 03 00:42:47.954571 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.954538 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:42:47.954671 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.954606 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:42:47.954671 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:47.954634 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:42:48.669955 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:48.669913 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" event={"ID":"3e6b2116-c767-4c94-867f-448307c7bb4f","Type":"ContainerStarted","Data":"81fcf90a30ff68df4c9c736c87db8aab9504c54c5e147b737a9624d79469df03"} Nov 03 00:42:48.670498 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:48.669986 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" event={"ID":"3e6b2116-c767-4c94-867f-448307c7bb4f","Type":"ContainerStarted","Data":"6276bbb512c7860176af6f14fa0bd18959a5ebcf3a33f90023ab96256ff14e62"} Nov 03 00:42:48.898848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:48.895870 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" podStartSLOduration=1.895847278 podStartE2EDuration="1.895847278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:48.835019377 +0000 UTC m=+2215.785959572" watchObservedRunningTime="2026-11-03 00:42:48.895847278 +0000 UTC m=+2215.846787436" Nov 03 00:42:49.526142 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:49.526108 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:49.527480 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:49.527453 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 00:42:49.527603 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:49.527504 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" podUID="3e6b2116-c767-4c94-867f-448307c7bb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 00:42:50.505751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:50.505617 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:50.532887 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:50.526853 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 00:42:50.532887 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:50.526917 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" podUID="3e6b2116-c767-4c94-867f-448307c7bb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 00:42:51.527052 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:51.527011 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 00:42:51.527537 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:51.527086 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" podUID="3e6b2116-c767-4c94-867f-448307c7bb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 00:42:52.527062 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:52.527027 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 00:42:52.527464 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:52.527079 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" podUID="3e6b2116-c767-4c94-867f-448307c7bb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 00:42:53.526713 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:53.526675 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 00:42:53.526922 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:53.526727 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" podUID="3e6b2116-c767-4c94-867f-448307c7bb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 00:42:54.526915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:54.526873 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 00:42:54.527324 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:54.526935 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" podUID="3e6b2116-c767-4c94-867f-448307c7bb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 00:42:55.583809 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:55.583765 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:55.584339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:55.584231 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:55.585436 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:55.585401 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:42:55.713243 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:55.713214 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:55.970981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:55.970893 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:57.154087 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:42:57.154049 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:42:58.722107 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:42:58.722073 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:00.384093 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:00.384062 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:04.509866 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:04.509814 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:05.147335 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:05.147290 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:05.557805 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:05.557763 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:05.856717 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:05.856624 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:09.574204 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:09.574167 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:10.498537 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:10.498504 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:11.092441 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:11.092399 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:11.413779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:11.413693 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:16.126092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:16.126052 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:16.778335 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:16.778254 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:16.949674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:16.949621 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:18.336246 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:18.336204 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:18.852008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:18.851970 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:20.639515 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:20.639478 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:22.492254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:22.492214 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:25.614144 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:25.614107 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:25.787755 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:25.787716 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:27.146940 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:27.146900 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:27.155596 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:43:27.155551 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:43:27.501888 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:27.501768 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:27.676339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:27.676300 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:32.506500 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:32.506409 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:32.675358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:32.675326 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:43:33.529737 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:33.529700 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:37.237620 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:37.237582 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:39.379141 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:39.379100 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:39.961799 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:39.961762 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9"] Nov 03 00:43:40.074393 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:40.074355 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x"] Nov 03 00:43:40.359947 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:40.359901 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:40.516986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:40.516930 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:42.424268 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:42.424228 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:44.963058 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:44.963007 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" podUID="d69c5308-3b24-401a-aab8-d3c60d85b690" containerName="payload-processing" containerID="cri-o://a67428e7c59ec0847aaf9263cd4a2229b3242affb25c152cd7624f6985785ad2" gracePeriod=25 Nov 03 00:43:45.075210 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.075152 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" podUID="d67b18c6-40ba-46e4-9e34-093a4dd6ec77" containerName="payload-pre-processing" containerID="cri-o://819563eb8f88a2d1a5690f8914c0e7ce6bd7ae07fa3b81cb89e9a5e498b8cc10" gracePeriod=25 Nov 03 00:43:45.086318 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.086292 6400 generic.go:356] "Generic (PLEG): container finished" podID="d69c5308-3b24-401a-aab8-d3c60d85b690" containerID="a67428e7c59ec0847aaf9263cd4a2229b3242affb25c152cd7624f6985785ad2" exitCode=0 Nov 03 00:43:45.086446 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.086366 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" event={"ID":"d69c5308-3b24-401a-aab8-d3c60d85b690","Type":"ContainerDied","Data":"a67428e7c59ec0847aaf9263cd4a2229b3242affb25c152cd7624f6985785ad2"} Nov 03 00:43:45.150022 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.149987 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:45.242040 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.242018 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" Nov 03 00:43:45.304019 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.303987 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69c5308-3b24-401a-aab8-d3c60d85b690-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69c5308-3b24-401a-aab8-d3c60d85b690-plugins-config-volume\") pod \"d69c5308-3b24-401a-aab8-d3c60d85b690\" (UID: \"d69c5308-3b24-401a-aab8-d3c60d85b690\") " Nov 03 00:43:45.304206 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.304039 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69c5308-3b24-401a-aab8-d3c60d85b690-kube-api-access-v4mtk\" (UniqueName: \"kubernetes.io/projected/d69c5308-3b24-401a-aab8-d3c60d85b690-kube-api-access-v4mtk\") pod \"d69c5308-3b24-401a-aab8-d3c60d85b690\" (UID: \"d69c5308-3b24-401a-aab8-d3c60d85b690\") " Nov 03 00:43:45.304394 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.304367 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69c5308-3b24-401a-aab8-d3c60d85b690-plugins-config-volume" pod "d69c5308-3b24-401a-aab8-d3c60d85b690" (UID: "d69c5308-3b24-401a-aab8-d3c60d85b690"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:45.306563 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.306542 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69c5308-3b24-401a-aab8-d3c60d85b690-kube-api-access-v4mtk" pod "d69c5308-3b24-401a-aab8-d3c60d85b690" (UID: "d69c5308-3b24-401a-aab8-d3c60d85b690"). InnerVolumeSpecName "kube-api-access-v4mtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:45.341698 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.341677 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" Nov 03 00:43:45.405075 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.405038 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-plugins-config-volume\") pod \"d67b18c6-40ba-46e4-9e34-093a4dd6ec77\" (UID: \"d67b18c6-40ba-46e4-9e34-093a4dd6ec77\") " Nov 03 00:43:45.405229 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.405144 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-kube-api-access-vtw4v\" (UniqueName: \"kubernetes.io/projected/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-kube-api-access-vtw4v\") pod \"d67b18c6-40ba-46e4-9e34-093a4dd6ec77\" (UID: \"d67b18c6-40ba-46e4-9e34-093a4dd6ec77\") " Nov 03 00:43:45.405483 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.405460 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-plugins-config-volume" pod "d67b18c6-40ba-46e4-9e34-093a4dd6ec77" (UID: "d67b18c6-40ba-46e4-9e34-093a4dd6ec77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:45.405598 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.405585 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-plugins-config-volume\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:45.405644 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.405601 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69c5308-3b24-401a-aab8-d3c60d85b690-plugins-config-volume\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:45.405644 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.405612 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4mtk\" (UniqueName: \"kubernetes.io/projected/d69c5308-3b24-401a-aab8-d3c60d85b690-kube-api-access-v4mtk\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:45.407361 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.407340 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-kube-api-access-vtw4v" pod "d67b18c6-40ba-46e4-9e34-093a4dd6ec77" (UID: "d67b18c6-40ba-46e4-9e34-093a4dd6ec77"). InnerVolumeSpecName "kube-api-access-vtw4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:45.506869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:45.506775 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtw4v\" (UniqueName: \"kubernetes.io/projected/d67b18c6-40ba-46e4-9e34-093a4dd6ec77-kube-api-access-vtw4v\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:46.093363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.093335 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" Nov 03 00:43:46.093851 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.093326 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9" event={"ID":"d69c5308-3b24-401a-aab8-d3c60d85b690","Type":"ContainerDied","Data":"fdf83e996cc1b781f0a1cdd74453fef8666b30e165fd2c17940ca912310fcfd7"} Nov 03 00:43:46.093851 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.093516 6400 scope.go:122] "RemoveContainer" containerID="a67428e7c59ec0847aaf9263cd4a2229b3242affb25c152cd7624f6985785ad2" Nov 03 00:43:46.094878 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.094573 6400 generic.go:356] "Generic (PLEG): container finished" podID="d67b18c6-40ba-46e4-9e34-093a4dd6ec77" containerID="819563eb8f88a2d1a5690f8914c0e7ce6bd7ae07fa3b81cb89e9a5e498b8cc10" exitCode=0 Nov 03 00:43:46.094878 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.094619 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" Nov 03 00:43:46.094878 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.094655 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" event={"ID":"d67b18c6-40ba-46e4-9e34-093a4dd6ec77","Type":"ContainerDied","Data":"819563eb8f88a2d1a5690f8914c0e7ce6bd7ae07fa3b81cb89e9a5e498b8cc10"} Nov 03 00:43:46.094878 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.094694 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x" event={"ID":"d67b18c6-40ba-46e4-9e34-093a4dd6ec77","Type":"ContainerDied","Data":"28a19e3add23856eaa2a6c8c358668e21d67cecdadca1aa20ae2fe86635bea23"} Nov 03 00:43:46.107187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.107060 6400 scope.go:122] "RemoveContainer" containerID="819563eb8f88a2d1a5690f8914c0e7ce6bd7ae07fa3b81cb89e9a5e498b8cc10" Nov 03 00:43:46.116538 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.116519 6400 scope.go:122] "RemoveContainer" containerID="819563eb8f88a2d1a5690f8914c0e7ce6bd7ae07fa3b81cb89e9a5e498b8cc10" Nov 03 00:43:46.116806 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:43:46.116774 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819563eb8f88a2d1a5690f8914c0e7ce6bd7ae07fa3b81cb89e9a5e498b8cc10\": container with ID starting with 819563eb8f88a2d1a5690f8914c0e7ce6bd7ae07fa3b81cb89e9a5e498b8cc10 not found: ID does not exist" containerID="819563eb8f88a2d1a5690f8914c0e7ce6bd7ae07fa3b81cb89e9a5e498b8cc10" Nov 03 00:43:46.116876 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.116843 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819563eb8f88a2d1a5690f8914c0e7ce6bd7ae07fa3b81cb89e9a5e498b8cc10"} err="failed to get container status \"819563eb8f88a2d1a5690f8914c0e7ce6bd7ae07fa3b81cb89e9a5e498b8cc10\": rpc error: code = NotFound desc = could not find container \"819563eb8f88a2d1a5690f8914c0e7ce6bd7ae07fa3b81cb89e9a5e498b8cc10\": container with ID starting with 819563eb8f88a2d1a5690f8914c0e7ce6bd7ae07fa3b81cb89e9a5e498b8cc10 not found: ID does not exist" Nov 03 00:43:46.255665 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.255570 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x"] Nov 03 00:43:46.305110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.305081 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d5db94-868c4964c-mpc8x"] Nov 03 00:43:46.367528 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.367495 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9"] Nov 03 00:43:46.371208 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:46.371176 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d5db94-7bd58bfd9-pl5k9"] Nov 03 00:43:47.178946 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:47.178913 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67b18c6-40ba-46e4-9e34-093a4dd6ec77" path="/var/lib/kubelet/pods/d67b18c6-40ba-46e4-9e34-093a4dd6ec77/volumes" Nov 03 00:43:47.179423 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:47.179316 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69c5308-3b24-401a-aab8-d3c60d85b690" path="/var/lib/kubelet/pods/d69c5308-3b24-401a-aab8-d3c60d85b690/volumes" Nov 03 00:43:47.194778 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:47.194739 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:51.901349 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:51.901312 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:56.553576 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:56.553537 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:57.152309 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:43:57.152269 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:43:59.762328 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:43:59.762292 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:44:01.217754 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:01.217720 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:06.119652 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:06.119616 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:10.831001 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:10.830970 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:11.438207 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:11.438163 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:14.909875 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:14.909841 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n"] Nov 03 00:44:14.910397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:14.910069 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" podUID="2b438591-e7cd-4569-af4f-95254774c42e" containerName="maas-api" containerID="cri-o://8be6a5a73936241a0c93f749a1c37254829c5ef5040033f0528b1b2e5c0f44ea" gracePeriod=30 Nov 03 00:44:14.996202 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:14.996169 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:15.230223 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.230198 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:44:15.266564 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.266532 6400 generic.go:356] "Generic (PLEG): container finished" podID="2b438591-e7cd-4569-af4f-95254774c42e" containerID="8be6a5a73936241a0c93f749a1c37254829c5ef5040033f0528b1b2e5c0f44ea" exitCode=0 Nov 03 00:44:15.266725 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.266580 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" Nov 03 00:44:15.266725 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.266625 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" event={"ID":"2b438591-e7cd-4569-af4f-95254774c42e","Type":"ContainerDied","Data":"8be6a5a73936241a0c93f749a1c37254829c5ef5040033f0528b1b2e5c0f44ea"} Nov 03 00:44:15.266725 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.266668 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n" event={"ID":"2b438591-e7cd-4569-af4f-95254774c42e","Type":"ContainerDied","Data":"ccb08cb22e405384d90d72df3240bf1baf8f644c0b1deb456f69c7da19a3fcac"} Nov 03 00:44:15.266725 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.266702 6400 scope.go:122] "RemoveContainer" containerID="8be6a5a73936241a0c93f749a1c37254829c5ef5040033f0528b1b2e5c0f44ea" Nov 03 00:44:15.276636 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.276617 6400 scope.go:122] "RemoveContainer" containerID="8be6a5a73936241a0c93f749a1c37254829c5ef5040033f0528b1b2e5c0f44ea" Nov 03 00:44:15.276983 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:44:15.276957 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be6a5a73936241a0c93f749a1c37254829c5ef5040033f0528b1b2e5c0f44ea\": container with ID starting with 8be6a5a73936241a0c93f749a1c37254829c5ef5040033f0528b1b2e5c0f44ea not found: ID does not exist" containerID="8be6a5a73936241a0c93f749a1c37254829c5ef5040033f0528b1b2e5c0f44ea" Nov 03 00:44:15.277075 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.276989 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be6a5a73936241a0c93f749a1c37254829c5ef5040033f0528b1b2e5c0f44ea"} err="failed to get container status \"8be6a5a73936241a0c93f749a1c37254829c5ef5040033f0528b1b2e5c0f44ea\": rpc error: code = NotFound desc = could not find container \"8be6a5a73936241a0c93f749a1c37254829c5ef5040033f0528b1b2e5c0f44ea\": container with ID starting with 8be6a5a73936241a0c93f749a1c37254829c5ef5040033f0528b1b2e5c0f44ea not found: ID does not exist" Nov 03 00:44:15.378135 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.378105 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-metrics-certs\") pod \"2b438591-e7cd-4569-af4f-95254774c42e\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " Nov 03 00:44:15.378325 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.378204 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b438591-e7cd-4569-af4f-95254774c42e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b438591-e7cd-4569-af4f-95254774c42e-openshift-service-ca\") pod \"2b438591-e7cd-4569-af4f-95254774c42e\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " Nov 03 00:44:15.378325 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.378252 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-maas-api-tls\") pod \"2b438591-e7cd-4569-af4f-95254774c42e\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " Nov 03 00:44:15.378466 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.378327 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b438591-e7cd-4569-af4f-95254774c42e-kube-api-access-plvft\" (UniqueName: \"kubernetes.io/projected/2b438591-e7cd-4569-af4f-95254774c42e-kube-api-access-plvft\") pod \"2b438591-e7cd-4569-af4f-95254774c42e\" (UID: \"2b438591-e7cd-4569-af4f-95254774c42e\") " Nov 03 00:44:15.378626 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.378599 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b438591-e7cd-4569-af4f-95254774c42e-openshift-service-ca" pod "2b438591-e7cd-4569-af4f-95254774c42e" (UID: "2b438591-e7cd-4569-af4f-95254774c42e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:44:15.378858 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.378839 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b438591-e7cd-4569-af4f-95254774c42e-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:15.380565 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.380539 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-metrics-certs" pod "2b438591-e7cd-4569-af4f-95254774c42e" (UID: "2b438591-e7cd-4569-af4f-95254774c42e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:44:15.380995 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.380970 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b438591-e7cd-4569-af4f-95254774c42e-kube-api-access-plvft" pod "2b438591-e7cd-4569-af4f-95254774c42e" (UID: "2b438591-e7cd-4569-af4f-95254774c42e"). InnerVolumeSpecName "kube-api-access-plvft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:15.381185 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.381163 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-maas-api-tls" pod "2b438591-e7cd-4569-af4f-95254774c42e" (UID: "2b438591-e7cd-4569-af4f-95254774c42e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:44:15.481798 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.481706 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:15.482063 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.482044 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b438591-e7cd-4569-af4f-95254774c42e-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:15.482178 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.482162 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plvft\" (UniqueName: \"kubernetes.io/projected/2b438591-e7cd-4569-af4f-95254774c42e-kube-api-access-plvft\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:15.686112 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.686078 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n"] Nov 03 00:44:15.691545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:15.691516 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d5db94-6967b544c5-k6l2n"] Nov 03 00:44:16.821369 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:16.821338 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:17.002958 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:17.002919 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:17.168497 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:17.168416 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:17.179285 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:17.179257 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b438591-e7cd-4569-af4f-95254774c42e" path="/var/lib/kubelet/pods/2b438591-e7cd-4569-af4f-95254774c42e/volumes" Nov 03 00:44:18.573434 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:18.572300 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:18.997761 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:18.997660 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq"] Nov 03 00:44:18.997995 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:18.997942 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" podUID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerName="istio-proxy" containerID="cri-o://b69f258218aa582cdcc67b2db750e1bbe7d706e777c9aef62a2cd7d91c099bca" gracePeriod=30 Nov 03 00:44:19.042998 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:19.042967 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:21.933116 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:21.933082 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:24.270998 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.270924 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:44:24.292018 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.291990 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-envoy\") pod \"15c3711f-87be-461b-a5d6-f7d6295b4002\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " Nov 03 00:44:24.292175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292056 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-certs\") pod \"15c3711f-87be-461b-a5d6-f7d6295b4002\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " Nov 03 00:44:24.292175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292086 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-token\" (UniqueName: \"kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-token\") pod \"15c3711f-87be-461b-a5d6-f7d6295b4002\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " Nov 03 00:44:24.292175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292137 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-kube-api-access-6pw7g\" (UniqueName: \"kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-kube-api-access-6pw7g\") pod \"15c3711f-87be-461b-a5d6-f7d6295b4002\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " Nov 03 00:44:24.292175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292172 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c3711f-87be-461b-a5d6-f7d6295b4002-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c3711f-87be-461b-a5d6-f7d6295b4002-istiod-ca-cert\") pod \"15c3711f-87be-461b-a5d6-f7d6295b4002\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " Nov 03 00:44:24.292384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292223 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-credential-socket\") pod \"15c3711f-87be-461b-a5d6-f7d6295b4002\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " Nov 03 00:44:24.292384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292270 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-data\") pod \"15c3711f-87be-461b-a5d6-f7d6295b4002\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " Nov 03 00:44:24.292384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292333 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-podinfo\") pod \"15c3711f-87be-461b-a5d6-f7d6295b4002\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " Nov 03 00:44:24.292384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292341 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-certs" pod "15c3711f-87be-461b-a5d6-f7d6295b4002" (UID: "15c3711f-87be-461b-a5d6-f7d6295b4002"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:44:24.292593 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292386 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-socket\") pod \"15c3711f-87be-461b-a5d6-f7d6295b4002\" (UID: \"15c3711f-87be-461b-a5d6-f7d6295b4002\") " Nov 03 00:44:24.292593 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292517 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-credential-socket" pod "15c3711f-87be-461b-a5d6-f7d6295b4002" (UID: "15c3711f-87be-461b-a5d6-f7d6295b4002"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:44:24.292702 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292605 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c3711f-87be-461b-a5d6-f7d6295b4002-istiod-ca-cert" pod "15c3711f-87be-461b-a5d6-f7d6295b4002" (UID: "15c3711f-87be-461b-a5d6-f7d6295b4002"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:44:24.292702 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292608 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-data" pod "15c3711f-87be-461b-a5d6-f7d6295b4002" (UID: "15c3711f-87be-461b-a5d6-f7d6295b4002"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:44:24.292957 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292938 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:24.293071 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292960 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c3711f-87be-461b-a5d6-f7d6295b4002-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:24.293071 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292970 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:24.293071 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292988 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:24.293071 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.292933 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-socket" pod "15c3711f-87be-461b-a5d6-f7d6295b4002" (UID: "15c3711f-87be-461b-a5d6-f7d6295b4002"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:44:24.294460 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.294438 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-envoy" pod "15c3711f-87be-461b-a5d6-f7d6295b4002" (UID: "15c3711f-87be-461b-a5d6-f7d6295b4002"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:44:24.294750 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.294728 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-kube-api-access-6pw7g" pod "15c3711f-87be-461b-a5d6-f7d6295b4002" (UID: "15c3711f-87be-461b-a5d6-f7d6295b4002"). InnerVolumeSpecName "kube-api-access-6pw7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:24.295044 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.295024 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-podinfo" pod "15c3711f-87be-461b-a5d6-f7d6295b4002" (UID: "15c3711f-87be-461b-a5d6-f7d6295b4002"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:44:24.295094 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.295047 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-token" pod "15c3711f-87be-461b-a5d6-f7d6295b4002" (UID: "15c3711f-87be-461b-a5d6-f7d6295b4002"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:24.318504 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.318470 6400 generic.go:356] "Generic (PLEG): container finished" podID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerID="b69f258218aa582cdcc67b2db750e1bbe7d706e777c9aef62a2cd7d91c099bca" exitCode=0 Nov 03 00:44:24.318632 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.318519 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" Nov 03 00:44:24.318632 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.318541 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" event={"ID":"15c3711f-87be-461b-a5d6-f7d6295b4002","Type":"ContainerDied","Data":"b69f258218aa582cdcc67b2db750e1bbe7d706e777c9aef62a2cd7d91c099bca"} Nov 03 00:44:24.318632 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.318591 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq" event={"ID":"15c3711f-87be-461b-a5d6-f7d6295b4002","Type":"ContainerDied","Data":"daa4b1b01dcbf6e404eb7c0a7add06f2fb893f3b75b5a414950b11003f34df8a"} Nov 03 00:44:24.318632 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.318614 6400 scope.go:122] "RemoveContainer" containerID="b69f258218aa582cdcc67b2db750e1bbe7d706e777c9aef62a2cd7d91c099bca" Nov 03 00:44:24.328526 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.328508 6400 scope.go:122] "RemoveContainer" containerID="3b65431c89dbdc12f7f50dba7b1abac2fbe9ad797cc5543afc0d87865955fdc9" Nov 03 00:44:24.337593 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.337575 6400 scope.go:122] "RemoveContainer" containerID="b69f258218aa582cdcc67b2db750e1bbe7d706e777c9aef62a2cd7d91c099bca" Nov 03 00:44:24.337880 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:44:24.337859 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69f258218aa582cdcc67b2db750e1bbe7d706e777c9aef62a2cd7d91c099bca\": container with ID starting with b69f258218aa582cdcc67b2db750e1bbe7d706e777c9aef62a2cd7d91c099bca not found: ID does not exist" containerID="b69f258218aa582cdcc67b2db750e1bbe7d706e777c9aef62a2cd7d91c099bca" Nov 03 00:44:24.337936 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.337891 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69f258218aa582cdcc67b2db750e1bbe7d706e777c9aef62a2cd7d91c099bca"} err="failed to get container status \"b69f258218aa582cdcc67b2db750e1bbe7d706e777c9aef62a2cd7d91c099bca\": rpc error: code = NotFound desc = could not find container \"b69f258218aa582cdcc67b2db750e1bbe7d706e777c9aef62a2cd7d91c099bca\": container with ID starting with b69f258218aa582cdcc67b2db750e1bbe7d706e777c9aef62a2cd7d91c099bca not found: ID does not exist" Nov 03 00:44:24.337936 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.337908 6400 scope.go:122] "RemoveContainer" containerID="3b65431c89dbdc12f7f50dba7b1abac2fbe9ad797cc5543afc0d87865955fdc9" Nov 03 00:44:24.338175 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:44:24.338154 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b65431c89dbdc12f7f50dba7b1abac2fbe9ad797cc5543afc0d87865955fdc9\": container with ID starting with 3b65431c89dbdc12f7f50dba7b1abac2fbe9ad797cc5543afc0d87865955fdc9 not found: ID does not exist" containerID="3b65431c89dbdc12f7f50dba7b1abac2fbe9ad797cc5543afc0d87865955fdc9" Nov 03 00:44:24.338239 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.338183 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b65431c89dbdc12f7f50dba7b1abac2fbe9ad797cc5543afc0d87865955fdc9"} err="failed to get container status \"3b65431c89dbdc12f7f50dba7b1abac2fbe9ad797cc5543afc0d87865955fdc9\": rpc error: code = NotFound desc = could not find container \"3b65431c89dbdc12f7f50dba7b1abac2fbe9ad797cc5543afc0d87865955fdc9\": container with ID starting with 3b65431c89dbdc12f7f50dba7b1abac2fbe9ad797cc5543afc0d87865955fdc9 not found: ID does not exist" Nov 03 00:44:24.394280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.394247 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:24.394280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.394273 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:24.394280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.394283 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:24.394533 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.394293 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:24.394533 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.394303 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pw7g\" (UniqueName: \"kubernetes.io/projected/15c3711f-87be-461b-a5d6-f7d6295b4002-kube-api-access-6pw7g\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:24.545949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.545913 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq"] Nov 03 00:44:24.551332 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:24.551293 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-d5db94-openshift-default-74985bfcdf-cz6zq"] Nov 03 00:44:25.178991 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:25.178955 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c3711f-87be-461b-a5d6-f7d6295b4002" path="/var/lib/kubelet/pods/15c3711f-87be-461b-a5d6-f7d6295b4002/volumes" Nov 03 00:44:26.800991 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:26.800948 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:27.152800 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:44:27.152716 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:44:31.630264 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:31.629116 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:32.730899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:32.730859 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:33.955411 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:33.955373 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:34.183691 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:34.183659 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:36.318859 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:36.318807 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:38.223441 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:38.223402 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:38.446399 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:38.446361 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:40.764609 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:40.764571 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:41.298938 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:41.298901 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:41.476574 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:41.476537 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:41.676955 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:41.676734 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:41.945408 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:41.945321 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:43.674341 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674309 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9"] Nov 03 00:44:43.674791 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674779 6400 cpu_manager.go:414] "containerMap: removing container" podUID="2b438591-e7cd-4569-af4f-95254774c42e" containerName="maas-api" Nov 03 00:44:43.674863 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674793 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b438591-e7cd-4569-af4f-95254774c42e" containerName="maas-api" Nov 03 00:44:43.674863 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674808 6400 cpu_manager.go:414] "containerMap: removing container" podUID="d67b18c6-40ba-46e4-9e34-093a4dd6ec77" containerName="payload-pre-processing" Nov 03 00:44:43.674863 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674813 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67b18c6-40ba-46e4-9e34-093a4dd6ec77" containerName="payload-pre-processing" Nov 03 00:44:43.674863 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674849 6400 cpu_manager.go:414] "containerMap: removing container" podUID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerName="istio-proxy" Nov 03 00:44:43.674863 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674856 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerName="istio-proxy" Nov 03 00:44:43.675054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674881 6400 cpu_manager.go:414] "containerMap: removing container" podUID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerName="istio-proxy" Nov 03 00:44:43.675054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674887 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerName="istio-proxy" Nov 03 00:44:43.675054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674896 6400 cpu_manager.go:414] "containerMap: removing container" podUID="d69c5308-3b24-401a-aab8-d3c60d85b690" containerName="payload-processing" Nov 03 00:44:43.675054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674901 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69c5308-3b24-401a-aab8-d3c60d85b690" containerName="payload-processing" Nov 03 00:44:43.675054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674975 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerName="istio-proxy" Nov 03 00:44:43.675054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674982 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69c5308-3b24-401a-aab8-d3c60d85b690" containerName="payload-processing" Nov 03 00:44:43.675054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674988 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b438591-e7cd-4569-af4f-95254774c42e" containerName="maas-api" Nov 03 00:44:43.675054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.674995 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67b18c6-40ba-46e4-9e34-093a4dd6ec77" containerName="payload-pre-processing" Nov 03 00:44:43.675054 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.675003 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c3711f-87be-461b-a5d6-f7d6295b4002" containerName="istio-proxy" Nov 03 00:44:43.679881 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.679815 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.722899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.722868 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh"] Nov 03 00:44:43.726538 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.726512 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.731368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.731343 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e28e1a-openshift-default-dockercfg-4zwkf\"" Nov 03 00:44:43.773746 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.773714 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9"] Nov 03 00:44:43.795905 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.795871 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh"] Nov 03 00:44:43.828306 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.828273 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d7bfc2-openshift-default-dockercfg-g56sm\"" Nov 03 00:44:43.830850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.830810 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.830942 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.830921 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-874zd\" (UniqueName: \"kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-kube-api-access-874zd\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.831040 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831022 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.831089 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831075 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.831159 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831144 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.831201 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831172 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpcg\" (UniqueName: \"kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-kube-api-access-hwpcg\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.831247 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831226 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.831286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831257 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.831286 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831280 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.831372 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831301 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.831448 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831425 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.831532 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831517 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.831595 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831551 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.831595 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831582 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.831679 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831658 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.831729 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831704 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.831794 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831778 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.831870 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.831850 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b6ff25-e381-44a1-afe6-9ff58381efe5-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.933552 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.933453 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.933552 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.933511 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.933814 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.933561 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.933814 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.933577 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpcg\" (UniqueName: \"kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-kube-api-access-hwpcg\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.933814 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.933598 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.933814 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.933617 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.933814 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.933648 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.933814 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.933763 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.934200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.933851 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.934200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.933917 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.934200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.933960 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.934200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.933979 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.934200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.933991 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.934200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.934016 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.934200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.934027 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.934200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.934037 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.934200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.934065 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.934200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.934084 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.934981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.934294 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.934981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.934339 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.934981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.934386 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.934981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.934421 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.934981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.934448 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b6ff25-e381-44a1-afe6-9ff58381efe5-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.934981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.934504 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.934981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.934563 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-874zd\" (UniqueName: \"kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-kube-api-access-874zd\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.935388 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.935177 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.935388 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.935198 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.935388 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.935350 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b6ff25-e381-44a1-afe6-9ff58381efe5-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.936417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.936384 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.936673 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.936654 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.936924 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.936902 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.937193 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.937177 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.991454 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.991427 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.991609 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.991431 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:43.991814 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.991795 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-874zd\" (UniqueName: \"kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-kube-api-access-874zd\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.992948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.992926 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:43.993237 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:43.993216 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpcg\" (UniqueName: \"kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-kube-api-access-hwpcg\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:44.039306 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:44.039278 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:44.431273 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:44.431249 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9"] Nov 03 00:44:44.445599 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:44.445555 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:44:44.445707 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:44.445642 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:44:44.445707 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:44.445684 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:44:44.447050 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:44.447031 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh"] Nov 03 00:44:44.462646 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:44:44.462609 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b6ff25_e381_44a1_afe6_9ff58381efe5.slice/crio-07b8c558fd18099407a134a2227d976bd0ee7829f70024540ca35524e88fbaed WatchSource:0}: Error finding container 07b8c558fd18099407a134a2227d976bd0ee7829f70024540ca35524e88fbaed: Status 404 returned error can't find the container with id 07b8c558fd18099407a134a2227d976bd0ee7829f70024540ca35524e88fbaed Nov 03 00:44:44.465063 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:44.465023 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:44:44.465136 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:44.465093 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:44:44.465136 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:44.465121 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:44:45.445270 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:45.445230 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" event={"ID":"f5f8a22b-db08-4df6-bfd6-e54789ac29e2","Type":"ContainerStarted","Data":"46d04b65d4c5fa88d1bfe027f19475563dfa56c519fcfec2d7abe141c7f9f78d"} Nov 03 00:44:45.445656 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:45.445287 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" event={"ID":"f5f8a22b-db08-4df6-bfd6-e54789ac29e2","Type":"ContainerStarted","Data":"ba612dab0a07b63bb23f95957af8470205327f73fa2830ff50df1418ad454365"} Nov 03 00:44:45.447674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:45.447639 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" event={"ID":"53b6ff25-e381-44a1-afe6-9ff58381efe5","Type":"ContainerStarted","Data":"0d1e5f120773375a366af96725423a030316213d33014f5396c6a3127c15c879"} Nov 03 00:44:45.447811 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:45.447682 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" event={"ID":"53b6ff25-e381-44a1-afe6-9ff58381efe5","Type":"ContainerStarted","Data":"07b8c558fd18099407a134a2227d976bd0ee7829f70024540ca35524e88fbaed"} Nov 03 00:44:45.671187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:45.671131 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" podStartSLOduration=2.6711130880000002 podStartE2EDuration="2.671113088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:45.621367458 +0000 UTC m=+2332.572307618" watchObservedRunningTime="2026-11-03 00:44:45.671113088 +0000 UTC m=+2332.622053245" Nov 03 00:44:45.740091 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:45.739984 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" podStartSLOduration=2.739966169 podStartE2EDuration="2.739966169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:45.739007901 +0000 UTC m=+2332.689948060" watchObservedRunningTime="2026-11-03 00:44:45.739966169 +0000 UTC m=+2332.690906393" Nov 03 00:44:45.999909 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:45.996463 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:45.999909 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:45.996600 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:44:45.999909 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:45.996642 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" podUID="f5f8a22b-db08-4df6-bfd6-e54789ac29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 00:44:46.039600 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:46.039566 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:46.040169 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:46.040147 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:46.041028 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:46.041005 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 00:44:46.041080 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:46.041049 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" podUID="53b6ff25-e381-44a1-afe6-9ff58381efe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 00:44:46.242074 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:46.242039 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:46.423405 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:46.423372 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:46.612188 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:46.612157 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:46.994283 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:46.994249 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:44:46.994583 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:46.994554 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" podUID="f5f8a22b-db08-4df6-bfd6-e54789ac29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 00:44:47.042253 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:47.042211 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 00:44:47.042466 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:47.042278 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" podUID="53b6ff25-e381-44a1-afe6-9ff58381efe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 00:44:47.997847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:47.997494 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:44:47.997847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:47.997573 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" podUID="f5f8a22b-db08-4df6-bfd6-e54789ac29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 00:44:48.041663 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:48.041616 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 00:44:48.041920 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:48.041892 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" podUID="53b6ff25-e381-44a1-afe6-9ff58381efe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 00:44:48.996844 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:48.995222 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:44:48.996844 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:48.995286 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" podUID="f5f8a22b-db08-4df6-bfd6-e54789ac29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 00:44:49.040594 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:49.040555 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 00:44:49.041047 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:49.040606 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" podUID="53b6ff25-e381-44a1-afe6-9ff58381efe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 00:44:49.427544 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:49.427508 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:49.612055 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:49.612018 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:49.997309 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:49.997269 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:44:50.030723 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:50.030685 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" podUID="f5f8a22b-db08-4df6-bfd6-e54789ac29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 00:44:50.040209 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:50.040135 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 00:44:50.042204 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:50.040509 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" podUID="53b6ff25-e381-44a1-afe6-9ff58381efe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 00:44:50.993853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:50.993801 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:44:50.994041 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:50.993875 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" podUID="f5f8a22b-db08-4df6-bfd6-e54789ac29e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 00:44:51.039700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:51.039667 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 00:44:51.039884 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:51.039741 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" podUID="53b6ff25-e381-44a1-afe6-9ff58381efe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 00:44:51.608402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:51.608364 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:52.047701 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:52.047656 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:52.048573 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:52.048546 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:52.052631 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:52.052604 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:44:52.185552 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:52.185515 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:52.186006 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:52.185983 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:52.187187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:52.187164 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:44:53.697806 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:53.697755 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:44:55.595449 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:55.595413 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:56.204921 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:56.204881 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:56.542057 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:56.542027 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:56.711122 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:56.711086 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:57.158736 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:44:57.158701 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:44:58.964447 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:44:58.964410 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:00.129964 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.129924 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:00.169266 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.169229 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj"] Nov 03 00:45:00.177233 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.177204 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" Nov 03 00:45:00.192446 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.192422 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx"] Nov 03 00:45:00.196291 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.196276 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" Nov 03 00:45:00.237905 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.237878 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:45:00.262814 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.262780 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3046c8-c876-4fbf-9a05-ee1161097c6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx\" (UID: \"3c3046c8-c876-4fbf-9a05-ee1161097c6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" Nov 03 00:45:00.262993 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.262922 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586424fe-2344-4e75-8b98-14bde2f732b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj\" (UID: \"586424fe-2344-4e75-8b98-14bde2f732b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" Nov 03 00:45:00.262993 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.262949 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42vrb\" (UniqueName: \"kubernetes.io/projected/3c3046c8-c876-4fbf-9a05-ee1161097c6e-kube-api-access-42vrb\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx\" (UID: \"3c3046c8-c876-4fbf-9a05-ee1161097c6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" Nov 03 00:45:00.263110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.263008 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpp2p\" (UniqueName: \"kubernetes.io/projected/586424fe-2344-4e75-8b98-14bde2f732b6-kube-api-access-kpp2p\") pod \"maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj\" (UID: \"586424fe-2344-4e75-8b98-14bde2f732b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" Nov 03 00:45:00.294984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.294943 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj"] Nov 03 00:45:00.367606 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.364458 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpp2p\" (UniqueName: \"kubernetes.io/projected/586424fe-2344-4e75-8b98-14bde2f732b6-kube-api-access-kpp2p\") pod \"maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj\" (UID: \"586424fe-2344-4e75-8b98-14bde2f732b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" Nov 03 00:45:00.367606 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.364591 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3046c8-c876-4fbf-9a05-ee1161097c6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx\" (UID: \"3c3046c8-c876-4fbf-9a05-ee1161097c6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" Nov 03 00:45:00.367606 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.364708 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586424fe-2344-4e75-8b98-14bde2f732b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj\" (UID: \"586424fe-2344-4e75-8b98-14bde2f732b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" Nov 03 00:45:00.367606 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.364745 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42vrb\" (UniqueName: \"kubernetes.io/projected/3c3046c8-c876-4fbf-9a05-ee1161097c6e-kube-api-access-42vrb\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx\" (UID: \"3c3046c8-c876-4fbf-9a05-ee1161097c6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" Nov 03 00:45:00.367606 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.366087 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3046c8-c876-4fbf-9a05-ee1161097c6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx\" (UID: \"3c3046c8-c876-4fbf-9a05-ee1161097c6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" Nov 03 00:45:00.367606 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.366758 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586424fe-2344-4e75-8b98-14bde2f732b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj\" (UID: \"586424fe-2344-4e75-8b98-14bde2f732b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" Nov 03 00:45:00.369210 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.369185 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx"] Nov 03 00:45:00.384847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.384739 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:00.422900 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.422867 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpp2p\" (UniqueName: \"kubernetes.io/projected/586424fe-2344-4e75-8b98-14bde2f732b6-kube-api-access-kpp2p\") pod \"maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj\" (UID: \"586424fe-2344-4e75-8b98-14bde2f732b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" Nov 03 00:45:00.423047 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.422981 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42vrb\" (UniqueName: \"kubernetes.io/projected/3c3046c8-c876-4fbf-9a05-ee1161097c6e-kube-api-access-42vrb\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx\" (UID: \"3c3046c8-c876-4fbf-9a05-ee1161097c6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" Nov 03 00:45:00.491385 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.491352 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" Nov 03 00:45:00.509057 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.509029 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" Nov 03 00:45:00.558745 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.558270 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:00.885003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.884977 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj"] Nov 03 00:45:00.888046 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:45:00.888009 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586424fe_2344_4e75_8b98_14bde2f732b6.slice/crio-3ac9cbbc849ad2e681c7208b7d6ef91544d45ef65e2cadea9b7d3638e328a9dd WatchSource:0}: Error finding container 3ac9cbbc849ad2e681c7208b7d6ef91544d45ef65e2cadea9b7d3638e328a9dd: Status 404 returned error can't find the container with id 3ac9cbbc849ad2e681c7208b7d6ef91544d45ef65e2cadea9b7d3638e328a9dd Nov 03 00:45:00.933724 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:00.933392 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx"] Nov 03 00:45:00.935860 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:45:00.935799 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3046c8_c876_4fbf_9a05_ee1161097c6e.slice/crio-e9f7a4eb4225664b02aea9f03dbe815016e44cadc1e44c583192a9f95eb8aa30 WatchSource:0}: Error finding container e9f7a4eb4225664b02aea9f03dbe815016e44cadc1e44c583192a9f95eb8aa30: Status 404 returned error can't find the container with id e9f7a4eb4225664b02aea9f03dbe815016e44cadc1e44c583192a9f95eb8aa30 Nov 03 00:45:01.594728 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:01.594693 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:01.648244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:01.646172 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" event={"ID":"3c3046c8-c876-4fbf-9a05-ee1161097c6e","Type":"ContainerStarted","Data":"1736c3a9354daf423302fbba986e7d0e43c2fe24d503571c62b01a825adba237"} Nov 03 00:45:01.648244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:01.646256 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" event={"ID":"3c3046c8-c876-4fbf-9a05-ee1161097c6e","Type":"ContainerStarted","Data":"e9f7a4eb4225664b02aea9f03dbe815016e44cadc1e44c583192a9f95eb8aa30"} Nov 03 00:45:01.650401 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:01.650360 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" event={"ID":"586424fe-2344-4e75-8b98-14bde2f732b6","Type":"ContainerStarted","Data":"878736bb544542486ba31a163dc41c0143db1b0f4b7cdb71a41e9d3ddd084c75"} Nov 03 00:45:01.650533 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:01.650417 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" event={"ID":"586424fe-2344-4e75-8b98-14bde2f732b6","Type":"ContainerStarted","Data":"3ac9cbbc849ad2e681c7208b7d6ef91544d45ef65e2cadea9b7d3638e328a9dd"} Nov 03 00:45:01.761564 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:01.761508 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" podStartSLOduration=1.761493822 podStartE2EDuration="1.761493822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:01.760503003 +0000 UTC m=+2348.711443163" watchObservedRunningTime="2026-11-03 00:45:01.761493822 +0000 UTC m=+2348.712434326" Nov 03 00:45:02.657999 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:02.657960 6400 generic.go:356] "Generic (PLEG): container finished" podID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" containerID="1736c3a9354daf423302fbba986e7d0e43c2fe24d503571c62b01a825adba237" exitCode=60 Nov 03 00:45:02.658458 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:02.658040 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" event={"ID":"3c3046c8-c876-4fbf-9a05-ee1161097c6e","Type":"ContainerDied","Data":"1736c3a9354daf423302fbba986e7d0e43c2fe24d503571c62b01a825adba237"} Nov 03 00:45:02.658458 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:02.658355 6400 scope.go:122] "RemoveContainer" containerID="1736c3a9354daf423302fbba986e7d0e43c2fe24d503571c62b01a825adba237" Nov 03 00:45:02.659673 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:02.659646 6400 generic.go:356] "Generic (PLEG): container finished" podID="586424fe-2344-4e75-8b98-14bde2f732b6" containerID="878736bb544542486ba31a163dc41c0143db1b0f4b7cdb71a41e9d3ddd084c75" exitCode=60 Nov 03 00:45:02.659799 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:02.659674 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" event={"ID":"586424fe-2344-4e75-8b98-14bde2f732b6","Type":"ContainerDied","Data":"878736bb544542486ba31a163dc41c0143db1b0f4b7cdb71a41e9d3ddd084c75"} Nov 03 00:45:02.660033 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:02.660015 6400 scope.go:122] "RemoveContainer" containerID="878736bb544542486ba31a163dc41c0143db1b0f4b7cdb71a41e9d3ddd084c75" Nov 03 00:45:02.772535 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:02.772487 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" podStartSLOduration=2.772472677 podStartE2EDuration="2.772472677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:01.884995088 +0000 UTC m=+2348.835935245" watchObservedRunningTime="2026-11-03 00:45:02.772472677 +0000 UTC m=+2349.723412835" Nov 03 00:45:03.431807 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:03.431768 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:03.667420 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:03.667379 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" event={"ID":"586424fe-2344-4e75-8b98-14bde2f732b6","Type":"ContainerStarted","Data":"28f92f4f8ff47fddc5b91fd1568ebe84a711bbd4a8d8f57c9d41259c577a8820"} Nov 03 00:45:03.669503 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:03.669474 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" event={"ID":"3c3046c8-c876-4fbf-9a05-ee1161097c6e","Type":"ContainerStarted","Data":"7395b766fcaf476e2e92b984792b1ea154ceb33203e8b927e598497c2d6bc326"} Nov 03 00:45:03.744316 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:03.744265 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:04.005090 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:04.005006 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:04.680562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:04.680528 6400 generic.go:356] "Generic (PLEG): container finished" podID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" containerID="7395b766fcaf476e2e92b984792b1ea154ceb33203e8b927e598497c2d6bc326" exitCode=60 Nov 03 00:45:04.681053 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:04.680611 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" event={"ID":"3c3046c8-c876-4fbf-9a05-ee1161097c6e","Type":"ContainerDied","Data":"7395b766fcaf476e2e92b984792b1ea154ceb33203e8b927e598497c2d6bc326"} Nov 03 00:45:04.681053 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:04.680666 6400 scope.go:122] "RemoveContainer" containerID="1736c3a9354daf423302fbba986e7d0e43c2fe24d503571c62b01a825adba237" Nov 03 00:45:04.681053 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:04.680919 6400 scope.go:122] "RemoveContainer" containerID="7395b766fcaf476e2e92b984792b1ea154ceb33203e8b927e598497c2d6bc326" Nov 03 00:45:04.681255 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:45:04.681233 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx_odh-ai-gateway-infra(3c3046c8-c876-4fbf-9a05-ee1161097c6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" podUID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" Nov 03 00:45:04.682728 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:04.682707 6400 generic.go:356] "Generic (PLEG): container finished" podID="586424fe-2344-4e75-8b98-14bde2f732b6" containerID="28f92f4f8ff47fddc5b91fd1568ebe84a711bbd4a8d8f57c9d41259c577a8820" exitCode=60 Nov 03 00:45:04.682809 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:04.682771 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" event={"ID":"586424fe-2344-4e75-8b98-14bde2f732b6","Type":"ContainerDied","Data":"28f92f4f8ff47fddc5b91fd1568ebe84a711bbd4a8d8f57c9d41259c577a8820"} Nov 03 00:45:04.683044 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:04.683030 6400 scope.go:122] "RemoveContainer" containerID="28f92f4f8ff47fddc5b91fd1568ebe84a711bbd4a8d8f57c9d41259c577a8820" Nov 03 00:45:04.683309 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:45:04.683287 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj_odh-ai-gateway-infra(586424fe-2344-4e75-8b98-14bde2f732b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" podUID="586424fe-2344-4e75-8b98-14bde2f732b6" Nov 03 00:45:04.692603 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:04.692585 6400 scope.go:122] "RemoveContainer" containerID="878736bb544542486ba31a163dc41c0143db1b0f4b7cdb71a41e9d3ddd084c75" Nov 03 00:45:05.690506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:05.690477 6400 scope.go:122] "RemoveContainer" containerID="7395b766fcaf476e2e92b984792b1ea154ceb33203e8b927e598497c2d6bc326" Nov 03 00:45:05.690953 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:45:05.690784 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx_odh-ai-gateway-infra(3c3046c8-c876-4fbf-9a05-ee1161097c6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" podUID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" Nov 03 00:45:05.692240 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:05.692210 6400 scope.go:122] "RemoveContainer" containerID="28f92f4f8ff47fddc5b91fd1568ebe84a711bbd4a8d8f57c9d41259c577a8820" Nov 03 00:45:05.692502 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:45:05.692481 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj_odh-ai-gateway-infra(586424fe-2344-4e75-8b98-14bde2f732b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" podUID="586424fe-2344-4e75-8b98-14bde2f732b6" Nov 03 00:45:06.952060 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:06.952022 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:09.112137 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:09.112104 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:09.728849 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:09.728794 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:09.954396 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:09.954356 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:11.729814 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:11.729774 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:11.902850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:11.902804 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:13.990020 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:13.989988 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:14.172065 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:14.172037 6400 scope.go:122] "RemoveContainer" containerID="28f92f4f8ff47fddc5b91fd1568ebe84a711bbd4a8d8f57c9d41259c577a8820" Nov 03 00:45:14.241388 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:14.241324 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:14.442999 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:14.442965 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:14.605747 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:14.605714 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:14.807901 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:14.807859 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" event={"ID":"586424fe-2344-4e75-8b98-14bde2f732b6","Type":"ContainerStarted","Data":"dbbe64d8b703eaff4eb8c57ba7bfea4bf4043b769b86b4bb0eed1cca1bfee9dc"} Nov 03 00:45:15.815713 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:15.815682 6400 generic.go:356] "Generic (PLEG): container finished" podID="586424fe-2344-4e75-8b98-14bde2f732b6" containerID="dbbe64d8b703eaff4eb8c57ba7bfea4bf4043b769b86b4bb0eed1cca1bfee9dc" exitCode=60 Nov 03 00:45:15.816143 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:15.815757 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" event={"ID":"586424fe-2344-4e75-8b98-14bde2f732b6","Type":"ContainerDied","Data":"dbbe64d8b703eaff4eb8c57ba7bfea4bf4043b769b86b4bb0eed1cca1bfee9dc"} Nov 03 00:45:15.816143 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:15.815846 6400 scope.go:122] "RemoveContainer" containerID="28f92f4f8ff47fddc5b91fd1568ebe84a711bbd4a8d8f57c9d41259c577a8820" Nov 03 00:45:15.816244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:15.816186 6400 scope.go:122] "RemoveContainer" containerID="dbbe64d8b703eaff4eb8c57ba7bfea4bf4043b769b86b4bb0eed1cca1bfee9dc" Nov 03 00:45:15.816481 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:45:15.816463 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj_odh-ai-gateway-infra(586424fe-2344-4e75-8b98-14bde2f732b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" podUID="586424fe-2344-4e75-8b98-14bde2f732b6" Nov 03 00:45:15.956425 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:15.956391 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj"] Nov 03 00:45:16.725352 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:16.725287 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:16.975420 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:16.975392 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" Nov 03 00:45:17.044734 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:17.044697 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586424fe-2344-4e75-8b98-14bde2f732b6-kube-api-access-kpp2p\" (UniqueName: \"kubernetes.io/projected/586424fe-2344-4e75-8b98-14bde2f732b6-kube-api-access-kpp2p\") pod \"586424fe-2344-4e75-8b98-14bde2f732b6\" (UID: \"586424fe-2344-4e75-8b98-14bde2f732b6\") " Nov 03 00:45:17.044958 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:17.044797 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/586424fe-2344-4e75-8b98-14bde2f732b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586424fe-2344-4e75-8b98-14bde2f732b6-maas-api-service-ca\") pod \"586424fe-2344-4e75-8b98-14bde2f732b6\" (UID: \"586424fe-2344-4e75-8b98-14bde2f732b6\") " Nov 03 00:45:17.045208 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:17.045181 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/586424fe-2344-4e75-8b98-14bde2f732b6-maas-api-service-ca" pod "586424fe-2344-4e75-8b98-14bde2f732b6" (UID: "586424fe-2344-4e75-8b98-14bde2f732b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:17.045356 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:17.045338 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586424fe-2344-4e75-8b98-14bde2f732b6-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:17.047479 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:17.047455 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586424fe-2344-4e75-8b98-14bde2f732b6-kube-api-access-kpp2p" pod "586424fe-2344-4e75-8b98-14bde2f732b6" (UID: "586424fe-2344-4e75-8b98-14bde2f732b6"). InnerVolumeSpecName "kube-api-access-kpp2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:17.146534 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:17.146448 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpp2p\" (UniqueName: \"kubernetes.io/projected/586424fe-2344-4e75-8b98-14bde2f732b6-kube-api-access-kpp2p\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:17.172235 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:17.172196 6400 scope.go:122] "RemoveContainer" containerID="7395b766fcaf476e2e92b984792b1ea154ceb33203e8b927e598497c2d6bc326" Nov 03 00:45:17.517322 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:17.517290 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:17.697276 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:17.697240 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:17.853914 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:17.853708 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" event={"ID":"3c3046c8-c876-4fbf-9a05-ee1161097c6e","Type":"ContainerStarted","Data":"a5c2f2f534f1ccbf788469f029a9522181e256e690cd5917b3e4c046788ece23"} Nov 03 00:45:17.856310 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:17.856285 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" Nov 03 00:45:17.856463 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:17.856440 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj" event={"ID":"586424fe-2344-4e75-8b98-14bde2f732b6","Type":"ContainerDied","Data":"3ac9cbbc849ad2e681c7208b7d6ef91544d45ef65e2cadea9b7d3638e328a9dd"} Nov 03 00:45:17.856571 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:17.856500 6400 scope.go:122] "RemoveContainer" containerID="dbbe64d8b703eaff4eb8c57ba7bfea4bf4043b769b86b4bb0eed1cca1bfee9dc" Nov 03 00:45:18.102945 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:18.102894 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj"] Nov 03 00:45:18.105159 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:18.105086 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-bdd194-29894445-x68sj"] Nov 03 00:45:18.864218 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:18.864187 6400 generic.go:356] "Generic (PLEG): container finished" podID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" containerID="a5c2f2f534f1ccbf788469f029a9522181e256e690cd5917b3e4c046788ece23" exitCode=60 Nov 03 00:45:18.864402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:18.864265 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" event={"ID":"3c3046c8-c876-4fbf-9a05-ee1161097c6e","Type":"ContainerDied","Data":"a5c2f2f534f1ccbf788469f029a9522181e256e690cd5917b3e4c046788ece23"} Nov 03 00:45:18.864402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:18.864334 6400 scope.go:122] "RemoveContainer" containerID="7395b766fcaf476e2e92b984792b1ea154ceb33203e8b927e598497c2d6bc326" Nov 03 00:45:18.864778 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:18.864748 6400 scope.go:122] "RemoveContainer" containerID="a5c2f2f534f1ccbf788469f029a9522181e256e690cd5917b3e4c046788ece23" Nov 03 00:45:18.865133 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:45:18.865110 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx_odh-ai-gateway-infra(3c3046c8-c876-4fbf-9a05-ee1161097c6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" podUID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" Nov 03 00:45:18.983623 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:18.983592 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx"] Nov 03 00:45:19.179551 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:19.179461 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="586424fe-2344-4e75-8b98-14bde2f732b6" path="/var/lib/kubelet/pods/586424fe-2344-4e75-8b98-14bde2f732b6/volumes" Nov 03 00:45:20.012087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:20.012057 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" Nov 03 00:45:20.120358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:20.120326 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3046c8-c876-4fbf-9a05-ee1161097c6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3046c8-c876-4fbf-9a05-ee1161097c6e-maas-api-service-ca\") pod \"3c3046c8-c876-4fbf-9a05-ee1161097c6e\" (UID: \"3c3046c8-c876-4fbf-9a05-ee1161097c6e\") " Nov 03 00:45:20.120555 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:20.120437 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3046c8-c876-4fbf-9a05-ee1161097c6e-kube-api-access-42vrb\" (UniqueName: \"kubernetes.io/projected/3c3046c8-c876-4fbf-9a05-ee1161097c6e-kube-api-access-42vrb\") pod \"3c3046c8-c876-4fbf-9a05-ee1161097c6e\" (UID: \"3c3046c8-c876-4fbf-9a05-ee1161097c6e\") " Nov 03 00:45:20.120711 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:20.120684 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3046c8-c876-4fbf-9a05-ee1161097c6e-maas-api-service-ca" pod "3c3046c8-c876-4fbf-9a05-ee1161097c6e" (UID: "3c3046c8-c876-4fbf-9a05-ee1161097c6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:20.120801 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:20.120789 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3046c8-c876-4fbf-9a05-ee1161097c6e-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:20.122770 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:20.122743 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3046c8-c876-4fbf-9a05-ee1161097c6e-kube-api-access-42vrb" pod "3c3046c8-c876-4fbf-9a05-ee1161097c6e" (UID: "3c3046c8-c876-4fbf-9a05-ee1161097c6e"). InnerVolumeSpecName "kube-api-access-42vrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:20.221793 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:20.221760 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42vrb\" (UniqueName: \"kubernetes.io/projected/3c3046c8-c876-4fbf-9a05-ee1161097c6e-kube-api-access-42vrb\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:20.880321 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:20.880287 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" Nov 03 00:45:20.880617 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:20.880325 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx" event={"ID":"3c3046c8-c876-4fbf-9a05-ee1161097c6e","Type":"ContainerDied","Data":"e9f7a4eb4225664b02aea9f03dbe815016e44cadc1e44c583192a9f95eb8aa30"} Nov 03 00:45:20.880617 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:20.880398 6400 scope.go:122] "RemoveContainer" containerID="a5c2f2f534f1ccbf788469f029a9522181e256e690cd5917b3e4c046788ece23" Nov 03 00:45:21.045308 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:21.045267 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx"] Nov 03 00:45:21.094914 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:21.094880 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894445-w45zx"] Nov 03 00:45:21.179340 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:21.179246 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" path="/var/lib/kubelet/pods/3c3046c8-c876-4fbf-9a05-ee1161097c6e/volumes" Nov 03 00:45:21.778937 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:21.778906 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:22.013932 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:22.013775 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:22.166465 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:22.166426 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:26.821741 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:26.821703 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:27.152631 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:45:27.152550 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:45:31.453665 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:31.453628 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:31.624271 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:31.624197 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:31.791089 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:31.791058 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:34.401718 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:34.401678 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:34.668415 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:34.668333 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:34.848020 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:34.838804 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:35.746844 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:35.744391 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:36.513437 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:36.513393 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:41.338544 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:41.338504 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:41.545882 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:41.545816 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:46.109469 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:46.109432 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:47.225486 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:47.225448 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:45:47.515903 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:47.507373 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:49.469781 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:49.469741 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:49.678782 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:49.678731 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:50.669062 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.669027 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp"] Nov 03 00:45:50.669541 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.669528 6400 cpu_manager.go:414] "containerMap: removing container" podUID="586424fe-2344-4e75-8b98-14bde2f732b6" containerName="cleanup" Nov 03 00:45:50.669589 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.669543 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="586424fe-2344-4e75-8b98-14bde2f732b6" containerName="cleanup" Nov 03 00:45:50.669589 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.669553 6400 cpu_manager.go:414] "containerMap: removing container" podUID="586424fe-2344-4e75-8b98-14bde2f732b6" containerName="cleanup" Nov 03 00:45:50.669589 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.669558 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="586424fe-2344-4e75-8b98-14bde2f732b6" containerName="cleanup" Nov 03 00:45:50.669589 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.669572 6400 cpu_manager.go:414] "containerMap: removing container" podUID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" containerName="cleanup" Nov 03 00:45:50.669589 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.669577 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" containerName="cleanup" Nov 03 00:45:50.669746 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.669606 6400 cpu_manager.go:414] "containerMap: removing container" podUID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" containerName="cleanup" Nov 03 00:45:50.669746 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.669611 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" containerName="cleanup" Nov 03 00:45:50.669746 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.669678 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" containerName="cleanup" Nov 03 00:45:50.669746 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.669688 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" containerName="cleanup" Nov 03 00:45:50.669746 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.669697 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="586424fe-2344-4e75-8b98-14bde2f732b6" containerName="cleanup" Nov 03 00:45:50.669746 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.669733 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="586424fe-2344-4e75-8b98-14bde2f732b6" containerName="cleanup" Nov 03 00:45:50.672952 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.672925 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:50.719383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.719353 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:50.719510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.719409 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:50.719510 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.719477 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctnb\" (UniqueName: \"kubernetes.io/projected/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-kube-api-access-sctnb\") pod \"maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:50.719591 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.719565 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:50.775835 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.775789 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-e4c8b1-serving-cert\"" Nov 03 00:45:50.778216 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.778197 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp"] Nov 03 00:45:50.821157 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.821125 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sctnb\" (UniqueName: \"kubernetes.io/projected/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-kube-api-access-sctnb\") pod \"maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:50.821341 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.821164 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:50.821341 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.821260 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:50.821468 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.821337 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:50.821468 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:45:50.821360 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-serving-cert: secret "maas-api-e2e-shared-a-w3-e4c8b1-serving-cert" not found Nov 03 00:45:50.821468 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:45:50.821422 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-maas-api-tls podName:5faaa2d3-41ef-42c6-a47c-8af1daeb5798 nodeName:}" failed. No retries permitted until 2026-11-03 00:45:51.321403491 +0000 UTC m=+2398.272343630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-maas-api-tls") pod "maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" (UID: "5faaa2d3-41ef-42c6-a47c-8af1daeb5798") : secret "maas-api-e2e-shared-a-w3-e4c8b1-serving-cert" not found Nov 03 00:45:50.822138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.822114 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:50.824062 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.824039 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:50.878102 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.878072 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctnb\" (UniqueName: \"kubernetes.io/projected/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-kube-api-access-sctnb\") pod \"maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:50.900732 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:50.900701 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:51.116689 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:51.116652 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:51.269212 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:51.269176 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:51.331045 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:51.331016 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:51.333675 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:51.333650 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:51.586380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:51.586348 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:51.966692 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:51.966630 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp"] Nov 03 00:45:51.969462 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:45:51.969433 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5faaa2d3_41ef_42c6_a47c_8af1daeb5798.slice/crio-7ddb082774027cee0a3fe5192e646c2a873adb99472b95b90a4137604a058abe WatchSource:0}: Error finding container 7ddb082774027cee0a3fe5192e646c2a873adb99472b95b90a4137604a058abe: Status 404 returned error can't find the container with id 7ddb082774027cee0a3fe5192e646c2a873adb99472b95b90a4137604a058abe Nov 03 00:45:51.971178 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:51.971158 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:45:52.058414 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:52.058383 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" event={"ID":"5faaa2d3-41ef-42c6-a47c-8af1daeb5798","Type":"ContainerStarted","Data":"7ddb082774027cee0a3fe5192e646c2a873adb99472b95b90a4137604a058abe"} Nov 03 00:45:53.065427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:53.065387 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" event={"ID":"5faaa2d3-41ef-42c6-a47c-8af1daeb5798","Type":"ContainerStarted","Data":"7fc22a35de1a3cb9af8ae0c018e233dcf8d550fb05392cc4e46fde76a993def3"} Nov 03 00:45:53.065938 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:53.065556 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:53.187371 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:53.187316 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" podStartSLOduration=2.393522795 podStartE2EDuration="3.187298075s" totalImagesPullingTime="793.77528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:53.184243771 +0000 UTC m=+2400.135183929" watchObservedRunningTime="2026-11-03 00:45:53.187298075 +0000 UTC m=+2400.138238232" Nov 03 00:45:55.856951 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:55.856919 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:57.154002 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:45:57.153969 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:45:59.078229 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:59.078203 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 00:45:59.940279 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:45:59.940237 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:01.658782 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:01.658748 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:02.570399 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:02.570362 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:02.757261 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:02.757222 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:04.260071 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:04.260025 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:04.720780 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:04.720750 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx"] Nov 03 00:46:04.721077 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:04.721049 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" containerID="cri-o://ccabfa5d6733f38ecd5ed18c4ddacfa340647b2f0b4397006452d660626f3bd4" gracePeriod=30 Nov 03 00:46:04.779193 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:04.779158 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:05.603776 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:05.603747 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:06.358003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:06.357973 6400 patch_prober.go:29] interesting pod/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:46:06.358203 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:06.358052 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:46:06.717161 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:06.717074 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:06.918842 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:06.918518 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:10.029962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.029938 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:46:10.165796 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.165765 6400 generic.go:356] "Generic (PLEG): container finished" podID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerID="ccabfa5d6733f38ecd5ed18c4ddacfa340647b2f0b4397006452d660626f3bd4" exitCode=0 Nov 03 00:46:10.165986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.165843 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" Nov 03 00:46:10.165986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.165853 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" event={"ID":"22f4272a-6597-4d6a-9f23-8562ba6944cd","Type":"ContainerDied","Data":"ccabfa5d6733f38ecd5ed18c4ddacfa340647b2f0b4397006452d660626f3bd4"} Nov 03 00:46:10.165986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.165914 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx" event={"ID":"22f4272a-6597-4d6a-9f23-8562ba6944cd","Type":"ContainerDied","Data":"3a4179cd40abda52b0c8dde2ca5b06c27c115734178499860ce7857224753702"} Nov 03 00:46:10.165986 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.165954 6400 scope.go:122] "RemoveContainer" containerID="ccabfa5d6733f38ecd5ed18c4ddacfa340647b2f0b4397006452d660626f3bd4" Nov 03 00:46:10.175089 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.175075 6400 scope.go:122] "RemoveContainer" containerID="cf3940076e8293246d5c5f2c9210bd140ef0cf1d83a40a23adb5ecf32142d063" Nov 03 00:46:10.186110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.186086 6400 scope.go:122] "RemoveContainer" containerID="ccabfa5d6733f38ecd5ed18c4ddacfa340647b2f0b4397006452d660626f3bd4" Nov 03 00:46:10.186412 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:46:10.186392 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccabfa5d6733f38ecd5ed18c4ddacfa340647b2f0b4397006452d660626f3bd4\": container with ID starting with ccabfa5d6733f38ecd5ed18c4ddacfa340647b2f0b4397006452d660626f3bd4 not found: ID does not exist" containerID="ccabfa5d6733f38ecd5ed18c4ddacfa340647b2f0b4397006452d660626f3bd4" Nov 03 00:46:10.186468 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.186431 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccabfa5d6733f38ecd5ed18c4ddacfa340647b2f0b4397006452d660626f3bd4"} err="failed to get container status \"ccabfa5d6733f38ecd5ed18c4ddacfa340647b2f0b4397006452d660626f3bd4\": rpc error: code = NotFound desc = could not find container \"ccabfa5d6733f38ecd5ed18c4ddacfa340647b2f0b4397006452d660626f3bd4\": container with ID starting with ccabfa5d6733f38ecd5ed18c4ddacfa340647b2f0b4397006452d660626f3bd4 not found: ID does not exist" Nov 03 00:46:10.186468 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.186448 6400 scope.go:122] "RemoveContainer" containerID="cf3940076e8293246d5c5f2c9210bd140ef0cf1d83a40a23adb5ecf32142d063" Nov 03 00:46:10.186738 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:46:10.186719 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3940076e8293246d5c5f2c9210bd140ef0cf1d83a40a23adb5ecf32142d063\": container with ID starting with cf3940076e8293246d5c5f2c9210bd140ef0cf1d83a40a23adb5ecf32142d063 not found: ID does not exist" containerID="cf3940076e8293246d5c5f2c9210bd140ef0cf1d83a40a23adb5ecf32142d063" Nov 03 00:46:10.186808 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.186743 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3940076e8293246d5c5f2c9210bd140ef0cf1d83a40a23adb5ecf32142d063"} err="failed to get container status \"cf3940076e8293246d5c5f2c9210bd140ef0cf1d83a40a23adb5ecf32142d063\": rpc error: code = NotFound desc = could not find container \"cf3940076e8293246d5c5f2c9210bd140ef0cf1d83a40a23adb5ecf32142d063\": container with ID starting with cf3940076e8293246d5c5f2c9210bd140ef0cf1d83a40a23adb5ecf32142d063 not found: ID does not exist" Nov 03 00:46:10.194466 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.194438 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-podinfo\") pod \"22f4272a-6597-4d6a-9f23-8562ba6944cd\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " Nov 03 00:46:10.194550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.194484 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-envoy\") pod \"22f4272a-6597-4d6a-9f23-8562ba6944cd\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " Nov 03 00:46:10.194550 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.194510 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-certs\") pod \"22f4272a-6597-4d6a-9f23-8562ba6944cd\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " Nov 03 00:46:10.194630 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.194601 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-data\") pod \"22f4272a-6597-4d6a-9f23-8562ba6944cd\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " Nov 03 00:46:10.194667 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.194634 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-kube-api-access-n45cd\" (UniqueName: \"kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-kube-api-access-n45cd\") pod \"22f4272a-6597-4d6a-9f23-8562ba6944cd\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " Nov 03 00:46:10.194712 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.194668 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-token\" (UniqueName: \"kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-token\") pod \"22f4272a-6597-4d6a-9f23-8562ba6944cd\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " Nov 03 00:46:10.194843 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.194801 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-socket\") pod \"22f4272a-6597-4d6a-9f23-8562ba6944cd\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " Nov 03 00:46:10.194919 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.194868 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-credential-socket\") pod \"22f4272a-6597-4d6a-9f23-8562ba6944cd\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " Nov 03 00:46:10.194919 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.194887 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-certs" pod "22f4272a-6597-4d6a-9f23-8562ba6944cd" (UID: "22f4272a-6597-4d6a-9f23-8562ba6944cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:46:10.194919 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.194903 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-data" pod "22f4272a-6597-4d6a-9f23-8562ba6944cd" (UID: "22f4272a-6597-4d6a-9f23-8562ba6944cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:46:10.194919 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.194915 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f4272a-6597-4d6a-9f23-8562ba6944cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f4272a-6597-4d6a-9f23-8562ba6944cd-istiod-ca-cert\") pod \"22f4272a-6597-4d6a-9f23-8562ba6944cd\" (UID: \"22f4272a-6597-4d6a-9f23-8562ba6944cd\") " Nov 03 00:46:10.195141 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.195082 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-socket" pod "22f4272a-6597-4d6a-9f23-8562ba6944cd" (UID: "22f4272a-6597-4d6a-9f23-8562ba6944cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:46:10.195141 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.195105 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-credential-socket" pod "22f4272a-6597-4d6a-9f23-8562ba6944cd" (UID: "22f4272a-6597-4d6a-9f23-8562ba6944cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:46:10.195252 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.195222 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f4272a-6597-4d6a-9f23-8562ba6944cd-istiod-ca-cert" pod "22f4272a-6597-4d6a-9f23-8562ba6944cd" (UID: "22f4272a-6597-4d6a-9f23-8562ba6944cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:46:10.195549 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.195531 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:10.195633 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.195553 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:10.195633 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.195567 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:10.195633 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.195581 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:10.195633 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.195594 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f4272a-6597-4d6a-9f23-8562ba6944cd-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:10.196952 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.196927 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-envoy" pod "22f4272a-6597-4d6a-9f23-8562ba6944cd" (UID: "22f4272a-6597-4d6a-9f23-8562ba6944cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:46:10.197051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.197028 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-kube-api-access-n45cd" pod "22f4272a-6597-4d6a-9f23-8562ba6944cd" (UID: "22f4272a-6597-4d6a-9f23-8562ba6944cd"). InnerVolumeSpecName "kube-api-access-n45cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:10.197234 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.197211 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-podinfo" pod "22f4272a-6597-4d6a-9f23-8562ba6944cd" (UID: "22f4272a-6597-4d6a-9f23-8562ba6944cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:46:10.197346 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.197290 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-token" pod "22f4272a-6597-4d6a-9f23-8562ba6944cd" (UID: "22f4272a-6597-4d6a-9f23-8562ba6944cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:10.296904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.296877 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n45cd\" (UniqueName: \"kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-kube-api-access-n45cd\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:10.296904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.296905 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:10.297096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.296915 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:10.297096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.296923 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f4272a-6597-4d6a-9f23-8562ba6944cd-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:10.585774 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.585700 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx"] Nov 03 00:46:10.635259 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:10.635226 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d61eae-openshift-default-858bf67789-52mwx"] Nov 03 00:46:11.178898 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:11.178867 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" path="/var/lib/kubelet/pods/22f4272a-6597-4d6a-9f23-8562ba6944cd/volumes" Nov 03 00:46:11.594369 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:11.594315 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:11.827991 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:11.827958 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:12.086048 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:12.086019 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476"] Nov 03 00:46:12.138709 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:12.138665 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:12.379379 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:12.379283 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:12.559370 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:12.559322 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:14.664090 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:14.664058 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:46:16.651494 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:16.651462 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:17.086743 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:17.086682 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" podUID="c34111ab-2498-45bc-bfad-f35a49cffd7f" containerName="payload-pre-processing" containerID="cri-o://d060ec3e8a9213251fa61f9ff98e1a5eb7e895e25f8b351f43377213dec6269c" gracePeriod=25 Nov 03 00:46:17.175090 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:17.175047 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:17.206649 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:17.206623 6400 generic.go:356] "Generic (PLEG): container finished" podID="c34111ab-2498-45bc-bfad-f35a49cffd7f" containerID="d060ec3e8a9213251fa61f9ff98e1a5eb7e895e25f8b351f43377213dec6269c" exitCode=0 Nov 03 00:46:17.206745 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:17.206692 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" event={"ID":"c34111ab-2498-45bc-bfad-f35a49cffd7f","Type":"ContainerDied","Data":"d060ec3e8a9213251fa61f9ff98e1a5eb7e895e25f8b351f43377213dec6269c"} Nov 03 00:46:17.349703 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:17.349640 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" Nov 03 00:46:17.459291 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:17.459260 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34111ab-2498-45bc-bfad-f35a49cffd7f-kube-api-access-cgptl\" (UniqueName: \"kubernetes.io/projected/c34111ab-2498-45bc-bfad-f35a49cffd7f-kube-api-access-cgptl\") pod \"c34111ab-2498-45bc-bfad-f35a49cffd7f\" (UID: \"c34111ab-2498-45bc-bfad-f35a49cffd7f\") " Nov 03 00:46:17.459472 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:17.459332 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34111ab-2498-45bc-bfad-f35a49cffd7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34111ab-2498-45bc-bfad-f35a49cffd7f-plugins-config-volume\") pod \"c34111ab-2498-45bc-bfad-f35a49cffd7f\" (UID: \"c34111ab-2498-45bc-bfad-f35a49cffd7f\") " Nov 03 00:46:17.459664 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:17.459642 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34111ab-2498-45bc-bfad-f35a49cffd7f-plugins-config-volume" pod "c34111ab-2498-45bc-bfad-f35a49cffd7f" (UID: "c34111ab-2498-45bc-bfad-f35a49cffd7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:46:17.462621 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:17.462590 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34111ab-2498-45bc-bfad-f35a49cffd7f-kube-api-access-cgptl" pod "c34111ab-2498-45bc-bfad-f35a49cffd7f" (UID: "c34111ab-2498-45bc-bfad-f35a49cffd7f"). InnerVolumeSpecName "kube-api-access-cgptl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:17.561217 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:17.561181 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgptl\" (UniqueName: \"kubernetes.io/projected/c34111ab-2498-45bc-bfad-f35a49cffd7f-kube-api-access-cgptl\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:17.561217 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:17.561213 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34111ab-2498-45bc-bfad-f35a49cffd7f-plugins-config-volume\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:18.213301 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:18.213272 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" Nov 03 00:46:18.213760 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:18.213308 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476" event={"ID":"c34111ab-2498-45bc-bfad-f35a49cffd7f","Type":"ContainerDied","Data":"ad08d6c0c154e840488bd7b5968ed96936dca20d1a2ad29ca291ec62fdddbe40"} Nov 03 00:46:18.213760 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:18.213371 6400 scope.go:122] "RemoveContainer" containerID="d060ec3e8a9213251fa61f9ff98e1a5eb7e895e25f8b351f43377213dec6269c" Nov 03 00:46:18.375861 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:18.375834 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476"] Nov 03 00:46:18.424834 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:18.424790 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bdd194-66cdfc9dc5-4g476"] Nov 03 00:46:19.179048 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:19.179016 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34111ab-2498-45bc-bfad-f35a49cffd7f" path="/var/lib/kubelet/pods/c34111ab-2498-45bc-bfad-f35a49cffd7f/volumes" Nov 03 00:46:21.604734 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:21.604697 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:25.520917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:25.520881 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:25.695255 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:25.695221 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:25.967405 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:25.967369 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:26.704021 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:26.698358 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:27.053600 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:27.053561 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:27.154214 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:46:27.154174 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:46:27.229079 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:27.229042 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:30.557757 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:30.557719 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:30.736343 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:30.736309 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:30.948369 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:30.948269 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:32.123065 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:32.123027 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:33.198520 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:33.198487 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:33.666177 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:33.666141 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:34.866845 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:34.866452 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:35.046961 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:35.046719 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:37.119259 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:37.119221 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:37.475718 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:37.475637 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:37.652690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:37.652628 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:39.050467 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:39.050429 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:39.226489 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:39.226455 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:39.401574 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:39.401491 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:39.585228 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:39.585195 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:42.265003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:42.264965 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:42.540088 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:42.539949 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:42.757197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:42.757159 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:47.511373 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:47.511333 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:52.411949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:52.411904 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:53.291375 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:53.291345 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:46:53.563147 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:53.563068 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:53.784647 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:53.784616 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:54.635842 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:54.635798 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:57.153144 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:46:57.153104 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:46:57.565089 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:57.565050 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:57.722974 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:57.722935 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:58.072880 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.072843 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:58.554799 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.554763 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz"] Nov 03 00:46:58.555474 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555446 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c34111ab-2498-45bc-bfad-f35a49cffd7f" containerName="payload-pre-processing" Nov 03 00:46:58.555474 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555472 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34111ab-2498-45bc-bfad-f35a49cffd7f" containerName="payload-pre-processing" Nov 03 00:46:58.555674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555512 6400 cpu_manager.go:414] "containerMap: removing container" podUID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" containerName="cleanup" Nov 03 00:46:58.555674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555520 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" containerName="cleanup" Nov 03 00:46:58.555674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555540 6400 cpu_manager.go:414] "containerMap: removing container" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" Nov 03 00:46:58.555674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555547 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" Nov 03 00:46:58.555674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555564 6400 cpu_manager.go:414] "containerMap: removing container" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" Nov 03 00:46:58.555674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555571 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" Nov 03 00:46:58.555674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555584 6400 cpu_manager.go:414] "containerMap: removing container" podUID="586424fe-2344-4e75-8b98-14bde2f732b6" containerName="cleanup" Nov 03 00:46:58.555674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555591 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="586424fe-2344-4e75-8b98-14bde2f732b6" containerName="cleanup" Nov 03 00:46:58.555939 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555719 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" Nov 03 00:46:58.555939 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555730 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3046c8-c876-4fbf-9a05-ee1161097c6e" containerName="cleanup" Nov 03 00:46:58.555939 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555738 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f4272a-6597-4d6a-9f23-8562ba6944cd" containerName="istio-proxy" Nov 03 00:46:58.555939 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555743 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34111ab-2498-45bc-bfad-f35a49cffd7f" containerName="payload-pre-processing" Nov 03 00:46:58.555939 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.555750 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="586424fe-2344-4e75-8b98-14bde2f732b6" containerName="cleanup" Nov 03 00:46:58.560251 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.560231 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.679011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.677740 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df412b7-1ae9-4f96-a92a-85e29370dabb-istiod-ca-cert\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.679011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.677850 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-envoy\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.679011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.677919 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w588v\" (UniqueName: \"kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-kube-api-access-w588v\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.679011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.677986 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-data\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.679011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.678017 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-podinfo\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.679011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.678051 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-certs\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.679011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.678137 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-credential-socket\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.679011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.678215 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-token\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.679011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.678245 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-socket\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.680419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.680364 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c87e2f-openshift-default-dockercfg-lx2qs\"" Nov 03 00:46:58.690423 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.690339 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:58.723887 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.709736 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz"] Nov 03 00:46:58.780892 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.780378 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-credential-socket\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.784008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.780969 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-token\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.784008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.781012 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-socket\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.784008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.781057 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df412b7-1ae9-4f96-a92a-85e29370dabb-istiod-ca-cert\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.784008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.781129 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-envoy\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.784008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.781192 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w588v\" (UniqueName: \"kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-kube-api-access-w588v\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.784008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.781265 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-data\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.784008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.781292 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-podinfo\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.784008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.781321 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-certs\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.784008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.781660 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-certs\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.784008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.780840 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-credential-socket\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.784008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.782257 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-socket\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.784008 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.782855 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df412b7-1ae9-4f96-a92a-85e29370dabb-istiod-ca-cert\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.786722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.786699 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-data\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.790316 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.790266 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-podinfo\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.790683 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.790662 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-envoy\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.861888 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.861423 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w588v\" (UniqueName: \"kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-kube-api-access-w588v\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.867583 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.867546 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-token\") pod \"e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.874300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.874235 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:46:58.928262 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:58.928229 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:59.306182 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:59.306153 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz"] Nov 03 00:46:59.309070 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:46:59.309042 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df412b7_1ae9_4f96_a92a_85e29370dabb.slice/crio-05ee826c84d9e2b546d38267ba86a0a70b47e101b72e32da30db223251ab7627 WatchSource:0}: Error finding container 05ee826c84d9e2b546d38267ba86a0a70b47e101b72e32da30db223251ab7627: Status 404 returned error can't find the container with id 05ee826c84d9e2b546d38267ba86a0a70b47e101b72e32da30db223251ab7627 Nov 03 00:46:59.311620 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:59.311585 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:46:59.311709 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:59.311652 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:46:59.311709 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:59.311683 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:46:59.472260 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:59.472222 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" event={"ID":"4df412b7-1ae9-4f96-a92a-85e29370dabb","Type":"ContainerStarted","Data":"b8bbf078a209709ab13ec512e64d21d1263f44d8a9bcad7b8bbae0e5b026411a"} Nov 03 00:46:59.472479 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:59.472286 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" event={"ID":"4df412b7-1ae9-4f96-a92a-85e29370dabb","Type":"ContainerStarted","Data":"05ee826c84d9e2b546d38267ba86a0a70b47e101b72e32da30db223251ab7627"} Nov 03 00:46:59.589695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:59.589581 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" podStartSLOduration=1.589561945 podStartE2EDuration="1.589561945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:59.587744467 +0000 UTC m=+2466.538684639" watchObservedRunningTime="2026-11-03 00:46:59.589561945 +0000 UTC m=+2466.540502103" Nov 03 00:46:59.874736 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:46:59.874649 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:47:00.880404 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:00.880371 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:47:01.485967 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:01.485936 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:47:01.486933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:01.486910 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:47:02.405740 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:02.405707 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:02.670937 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:02.670815 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:05.667142 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:05.667093 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:05.855760 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:05.855721 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:06.858754 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:06.858715 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt"] Nov 03 00:47:06.859617 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:06.859582 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" podUID="43cf9519-081f-4426-a37a-789cb95c4a4a" containerName="maas-api" containerID="cri-o://d74f17e703c6ff28cbb5edff03664df641036431cf74f9e30eaf3f76391eeede" gracePeriod=30 Nov 03 00:47:06.966956 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:06.966920 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:07.005521 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.005440 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" podUID="43cf9519-081f-4426-a37a-789cb95c4a4a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.96:8443/health\": dial tcp 10.134.0.96:8443: connect: connection refused" Nov 03 00:47:07.234939 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.234910 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:47:07.305848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.305241 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-maas-api-tls\") pod \"43cf9519-081f-4426-a37a-789cb95c4a4a\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " Nov 03 00:47:07.305848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.305300 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43cf9519-081f-4426-a37a-789cb95c4a4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cf9519-081f-4426-a37a-789cb95c4a4a-openshift-service-ca\") pod \"43cf9519-081f-4426-a37a-789cb95c4a4a\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " Nov 03 00:47:07.305848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.305353 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cf9519-081f-4426-a37a-789cb95c4a4a-kube-api-access-ht5cd\" (UniqueName: \"kubernetes.io/projected/43cf9519-081f-4426-a37a-789cb95c4a4a-kube-api-access-ht5cd\") pod \"43cf9519-081f-4426-a37a-789cb95c4a4a\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " Nov 03 00:47:07.305848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.305470 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-metrics-certs\") pod \"43cf9519-081f-4426-a37a-789cb95c4a4a\" (UID: \"43cf9519-081f-4426-a37a-789cb95c4a4a\") " Nov 03 00:47:07.310075 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.310041 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cf9519-081f-4426-a37a-789cb95c4a4a-openshift-service-ca" pod "43cf9519-081f-4426-a37a-789cb95c4a4a" (UID: "43cf9519-081f-4426-a37a-789cb95c4a4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:07.313397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.313340 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-maas-api-tls" pod "43cf9519-081f-4426-a37a-789cb95c4a4a" (UID: "43cf9519-081f-4426-a37a-789cb95c4a4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:07.323962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.320025 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cf9519-081f-4426-a37a-789cb95c4a4a-kube-api-access-ht5cd" pod "43cf9519-081f-4426-a37a-789cb95c4a4a" (UID: "43cf9519-081f-4426-a37a-789cb95c4a4a"). InnerVolumeSpecName "kube-api-access-ht5cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:07.323962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.320155 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-metrics-certs" pod "43cf9519-081f-4426-a37a-789cb95c4a4a" (UID: "43cf9519-081f-4426-a37a-789cb95c4a4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:07.407282 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.407196 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht5cd\" (UniqueName: \"kubernetes.io/projected/43cf9519-081f-4426-a37a-789cb95c4a4a-kube-api-access-ht5cd\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:07.407282 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.407229 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:07.407282 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.407242 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cf9519-081f-4426-a37a-789cb95c4a4a-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:07.407282 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.407254 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cf9519-081f-4426-a37a-789cb95c4a4a-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:07.520992 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.520950 6400 generic.go:356] "Generic (PLEG): container finished" podID="43cf9519-081f-4426-a37a-789cb95c4a4a" containerID="d74f17e703c6ff28cbb5edff03664df641036431cf74f9e30eaf3f76391eeede" exitCode=0 Nov 03 00:47:07.521197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.521013 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" Nov 03 00:47:07.521197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.521040 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" event={"ID":"43cf9519-081f-4426-a37a-789cb95c4a4a","Type":"ContainerDied","Data":"d74f17e703c6ff28cbb5edff03664df641036431cf74f9e30eaf3f76391eeede"} Nov 03 00:47:07.521197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.521095 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt" event={"ID":"43cf9519-081f-4426-a37a-789cb95c4a4a","Type":"ContainerDied","Data":"ff34f538900316d060b5659906b912d43a71c26a954ed720fd6e2d612727b384"} Nov 03 00:47:07.521197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.521118 6400 scope.go:122] "RemoveContainer" containerID="d74f17e703c6ff28cbb5edff03664df641036431cf74f9e30eaf3f76391eeede" Nov 03 00:47:07.531344 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.531327 6400 scope.go:122] "RemoveContainer" containerID="d74f17e703c6ff28cbb5edff03664df641036431cf74f9e30eaf3f76391eeede" Nov 03 00:47:07.531649 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:47:07.531630 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74f17e703c6ff28cbb5edff03664df641036431cf74f9e30eaf3f76391eeede\": container with ID starting with d74f17e703c6ff28cbb5edff03664df641036431cf74f9e30eaf3f76391eeede not found: ID does not exist" containerID="d74f17e703c6ff28cbb5edff03664df641036431cf74f9e30eaf3f76391eeede" Nov 03 00:47:07.531700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.531660 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74f17e703c6ff28cbb5edff03664df641036431cf74f9e30eaf3f76391eeede"} err="failed to get container status \"d74f17e703c6ff28cbb5edff03664df641036431cf74f9e30eaf3f76391eeede\": rpc error: code = NotFound desc = could not find container \"d74f17e703c6ff28cbb5edff03664df641036431cf74f9e30eaf3f76391eeede\": container with ID starting with d74f17e703c6ff28cbb5edff03664df641036431cf74f9e30eaf3f76391eeede not found: ID does not exist" Nov 03 00:47:07.684271 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.684186 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt"] Nov 03 00:47:07.687705 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:07.687676 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bdd194-7df668975f-qbbjt"] Nov 03 00:47:08.970948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:08.970908 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:09.179097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:09.179062 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43cf9519-081f-4426-a37a-789cb95c4a4a" path="/var/lib/kubelet/pods/43cf9519-081f-4426-a37a-789cb95c4a4a/volumes" Nov 03 00:47:11.816344 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:11.816305 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:12.280889 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:12.280782 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:12.491175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:12.491133 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:13.099833 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.099783 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk"] Nov 03 00:47:13.100496 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.100477 6400 cpu_manager.go:414] "containerMap: removing container" podUID="43cf9519-081f-4426-a37a-789cb95c4a4a" containerName="maas-api" Nov 03 00:47:13.100548 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.100498 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43cf9519-081f-4426-a37a-789cb95c4a4a" containerName="maas-api" Nov 03 00:47:13.100610 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.100601 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="43cf9519-081f-4426-a37a-789cb95c4a4a" containerName="maas-api" Nov 03 00:47:13.105244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.105225 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.157196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.157163 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-20f0a3-openshift-default-dockercfg-9qfww\"" Nov 03 00:47:13.182339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.182304 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728c90f9-2248-493e-adc6-7e51e412154d-istio-podinfo\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.182524 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.182348 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-data\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.182524 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.182465 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-istio-token\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.182524 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.182492 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p62zw\" (UniqueName: \"kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-kube-api-access-p62zw\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.182673 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.182545 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-credential-socket\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.182673 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.182633 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-socket\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.182753 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.182722 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728c90f9-2248-493e-adc6-7e51e412154d-istiod-ca-cert\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.182800 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.182783 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-envoy\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.182878 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.182863 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-certs\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.209438 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.209402 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk"] Nov 03 00:47:13.234038 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.233995 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:13.284300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.284259 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728c90f9-2248-493e-adc6-7e51e412154d-istio-podinfo\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.284300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.284301 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-data\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.284580 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.284360 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-istio-token\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.284580 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.284380 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p62zw\" (UniqueName: \"kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-kube-api-access-p62zw\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.284690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.284611 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-credential-socket\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.284690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.284648 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-socket\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.284810 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.284770 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728c90f9-2248-493e-adc6-7e51e412154d-istiod-ca-cert\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.284897 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.284866 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-data\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.285001 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.284978 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-envoy\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.285109 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.285089 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-certs\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.285172 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.285129 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-socket\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.285557 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.285399 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-credential-socket\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.285557 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.285499 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728c90f9-2248-493e-adc6-7e51e412154d-istiod-ca-cert\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.285557 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.285538 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-certs\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.287100 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.287075 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728c90f9-2248-493e-adc6-7e51e412154d-istio-podinfo\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.287394 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.287375 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-envoy\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.351974 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.350607 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-istio-token\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.364843 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.364025 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p62zw\" (UniqueName: \"kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-kube-api-access-p62zw\") pod \"e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.408846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.399729 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:13.418543 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.418262 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:13.845838 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.845793 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk"] Nov 03 00:47:13.847991 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:47:13.847959 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728c90f9_2248_493e_adc6_7e51e412154d.slice/crio-a4365e80d15f312c90120e66f9b527fa970a46b27376ac5b1e8887e9209ec36c WatchSource:0}: Error finding container a4365e80d15f312c90120e66f9b527fa970a46b27376ac5b1e8887e9209ec36c: Status 404 returned error can't find the container with id a4365e80d15f312c90120e66f9b527fa970a46b27376ac5b1e8887e9209ec36c Nov 03 00:47:13.851099 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.851058 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:47:13.851204 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.851150 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:47:13.851204 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:13.851191 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:47:14.101176 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:14.101091 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:14.563738 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:14.563698 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" event={"ID":"728c90f9-2248-493e-adc6-7e51e412154d","Type":"ContainerStarted","Data":"67935ffc9efc96bfe0d96a47b96cc311f1223647cb8edc5bb3f2b40d03a9dca3"} Nov 03 00:47:14.563941 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:14.563789 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" event={"ID":"728c90f9-2248-493e-adc6-7e51e412154d","Type":"ContainerStarted","Data":"a4365e80d15f312c90120e66f9b527fa970a46b27376ac5b1e8887e9209ec36c"} Nov 03 00:47:14.729246 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:14.729193 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" podStartSLOduration=1.7291794820000002 podStartE2EDuration="1.729179482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:14.728251775 +0000 UTC m=+2481.679191934" watchObservedRunningTime="2026-11-03 00:47:14.729179482 +0000 UTC m=+2481.680119639" Nov 03 00:47:15.419852 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:15.419789 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:15.425462 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:15.425438 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:15.571838 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:15.571799 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:15.572685 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:15.572668 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:47:16.020587 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:16.020552 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:16.184619 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:16.184583 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:17.510563 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:17.510527 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:18.005676 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:18.005641 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:18.198448 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:18.198414 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:18.373895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:18.373863 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:19.129557 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:19.129527 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:21.305287 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:21.305248 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:21.515993 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:21.515959 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:24.192173 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:24.192137 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:24.364446 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:24.364406 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:24.523103 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:24.523023 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:26.604001 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:26.603960 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:27.164249 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:47:27.164207 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:47:29.207952 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:29.207911 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:33.570148 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.570111 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz"] Nov 03 00:47:33.574107 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.574087 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:33.677256 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.677228 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-328e1b-serving-cert\"" Nov 03 00:47:33.679562 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.679541 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz"] Nov 03 00:47:33.715076 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.715043 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:33.715242 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.715164 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4wbg\" (UniqueName: \"kubernetes.io/projected/29761a2e-1646-4996-9f3e-89a0ace1588d-kube-api-access-q4wbg\") pod \"maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:33.715242 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.715233 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29761a2e-1646-4996-9f3e-89a0ace1588d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:33.715431 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.715408 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:33.787519 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.787483 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:33.827700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.827608 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:33.828200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.828177 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:33.829001 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.828975 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4wbg\" (UniqueName: \"kubernetes.io/projected/29761a2e-1646-4996-9f3e-89a0ace1588d-kube-api-access-q4wbg\") pod \"maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:33.829136 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.829082 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29761a2e-1646-4996-9f3e-89a0ace1588d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:33.830198 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:47:33.828057 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-serving-cert: secret "maas-api-e2e-shared-b-w2-328e1b-serving-cert" not found Nov 03 00:47:33.830393 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:47:33.830381 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-maas-api-tls podName:29761a2e-1646-4996-9f3e-89a0ace1588d nodeName:}" failed. No retries permitted until 2026-11-03 00:47:34.330355122 +0000 UTC m=+2501.281295275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-maas-api-tls") pod "maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" (UID: "29761a2e-1646-4996-9f3e-89a0ace1588d") : secret "maas-api-e2e-shared-b-w2-328e1b-serving-cert" not found Nov 03 00:47:33.830908 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.830887 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29761a2e-1646-4996-9f3e-89a0ace1588d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:33.856852 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.852777 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:33.886125 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.886097 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4wbg\" (UniqueName: \"kubernetes.io/projected/29761a2e-1646-4996-9f3e-89a0ace1588d-kube-api-access-q4wbg\") pod \"maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:33.947660 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:33.947623 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:34.107209 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:34.107118 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:34.334954 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:34.334919 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:34.337636 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:34.337603 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:34.488337 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:34.488263 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:34.874404 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:34.874378 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz"] Nov 03 00:47:34.876913 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:47:34.876874 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29761a2e_1646_4996_9f3e_89a0ace1588d.slice/crio-5b23848bbbfdcdb8995e53c24a55328ed027142c0be22378965d3b360f68758d WatchSource:0}: Error finding container 5b23848bbbfdcdb8995e53c24a55328ed027142c0be22378965d3b360f68758d: Status 404 returned error can't find the container with id 5b23848bbbfdcdb8995e53c24a55328ed027142c0be22378965d3b360f68758d Nov 03 00:47:35.717612 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:35.717574 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" event={"ID":"29761a2e-1646-4996-9f3e-89a0ace1588d","Type":"ContainerStarted","Data":"5b23848bbbfdcdb8995e53c24a55328ed027142c0be22378965d3b360f68758d"} Nov 03 00:47:36.724596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:36.724559 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" event={"ID":"29761a2e-1646-4996-9f3e-89a0ace1588d","Type":"ContainerStarted","Data":"e3354647da10f814071d995e89732b14d7f02acd6c4861ebd5f8ea2f2e159788"} Nov 03 00:47:36.725022 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:36.724627 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:36.845333 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:36.845276 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" podStartSLOduration=2.9914992099999997 podStartE2EDuration="3.845261053s" totalImagesPullingTime="853.761843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:36.843259474 +0000 UTC m=+2503.794199633" watchObservedRunningTime="2026-11-03 00:47:36.845261053 +0000 UTC m=+2503.796201211" Nov 03 00:47:38.715102 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:38.715067 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:38.974655 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:38.974573 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:47:42.741749 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:42.741721 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 00:47:43.520407 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:43.520371 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:43.641085 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:43.641047 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n"] Nov 03 00:47:43.641287 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:43.641262 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" podUID="6dd0c634-3b18-4b15-8438-6cf01db8b352" containerName="istio-proxy" containerID="cri-o://f019f7cc6e5881b12d8a6d94f1429b84a237c2cfe9fd4a29e30c5ba0c55977f9" gracePeriod=30 Nov 03 00:47:43.690152 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:43.690123 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw"] Nov 03 00:47:43.690347 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:43.690326 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerName="istio-proxy" containerID="cri-o://6195135f9b7f24909ce27c8c217d0e950927c6d4aac3ddd2af33ae7358e47f7d" gracePeriod=30 Nov 03 00:47:46.772404 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:46.772365 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:47:46.772834 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:46.772435 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" podUID="6dd0c634-3b18-4b15-8438-6cf01db8b352" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:47:48.294930 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:48.294890 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:48.831566 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:48.831525 6400 generic.go:356] "Generic (PLEG): container finished" podID="6dd0c634-3b18-4b15-8438-6cf01db8b352" containerID="f019f7cc6e5881b12d8a6d94f1429b84a237c2cfe9fd4a29e30c5ba0c55977f9" exitCode=0 Nov 03 00:47:48.831751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:48.831589 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" event={"ID":"6dd0c634-3b18-4b15-8438-6cf01db8b352","Type":"ContainerDied","Data":"f019f7cc6e5881b12d8a6d94f1429b84a237c2cfe9fd4a29e30c5ba0c55977f9"} Nov 03 00:47:48.833219 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:48.833195 6400 generic.go:356] "Generic (PLEG): container finished" podID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerID="6195135f9b7f24909ce27c8c217d0e950927c6d4aac3ddd2af33ae7358e47f7d" exitCode=0 Nov 03 00:47:48.833334 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:48.833270 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" event={"ID":"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf","Type":"ContainerDied","Data":"6195135f9b7f24909ce27c8c217d0e950927c6d4aac3ddd2af33ae7358e47f7d"} Nov 03 00:47:48.959153 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:48.959122 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:47:48.989283 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:48.989262 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:47:49.015591 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.015566 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-credential-socket\") pod \"6dd0c634-3b18-4b15-8438-6cf01db8b352\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " Nov 03 00:47:49.015720 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.015609 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-certs\") pod \"6dd0c634-3b18-4b15-8438-6cf01db8b352\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " Nov 03 00:47:49.015720 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.015648 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd0c634-3b18-4b15-8438-6cf01db8b352-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd0c634-3b18-4b15-8438-6cf01db8b352-istiod-ca-cert\") pod \"6dd0c634-3b18-4b15-8438-6cf01db8b352\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " Nov 03 00:47:49.015720 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.015684 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-podinfo\") pod \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " Nov 03 00:47:49.015915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.015716 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-socket\") pod \"6dd0c634-3b18-4b15-8438-6cf01db8b352\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " Nov 03 00:47:49.015915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.015784 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-podinfo\") pod \"6dd0c634-3b18-4b15-8438-6cf01db8b352\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " Nov 03 00:47:49.015915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.015813 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-certs\") pod \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " Nov 03 00:47:49.015915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.015848 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-credential-socket" pod "6dd0c634-3b18-4b15-8438-6cf01db8b352" (UID: "6dd0c634-3b18-4b15-8438-6cf01db8b352"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:49.015915 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.015881 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-token\" (UniqueName: \"kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-token\") pod \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " Nov 03 00:47:49.016175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.015915 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-kube-api-access-zc8k8\" (UniqueName: \"kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-kube-api-access-zc8k8\") pod \"6dd0c634-3b18-4b15-8438-6cf01db8b352\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " Nov 03 00:47:49.016175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.015967 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-envoy\") pod \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " Nov 03 00:47:49.016175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.015979 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-certs" pod "6dd0c634-3b18-4b15-8438-6cf01db8b352" (UID: "6dd0c634-3b18-4b15-8438-6cf01db8b352"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:49.016175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.016027 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-envoy\") pod \"6dd0c634-3b18-4b15-8438-6cf01db8b352\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " Nov 03 00:47:49.016175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.016043 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-socket" pod "6dd0c634-3b18-4b15-8438-6cf01db8b352" (UID: "6dd0c634-3b18-4b15-8438-6cf01db8b352"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:49.016175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.016059 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-token\" (UniqueName: \"kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-token\") pod \"6dd0c634-3b18-4b15-8438-6cf01db8b352\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " Nov 03 00:47:49.016175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.016079 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd0c634-3b18-4b15-8438-6cf01db8b352-istiod-ca-cert" pod "6dd0c634-3b18-4b15-8438-6cf01db8b352" (UID: "6dd0c634-3b18-4b15-8438-6cf01db8b352"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:49.016175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.016116 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istiod-ca-cert\") pod \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " Nov 03 00:47:49.016175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.016149 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-socket\") pod \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " Nov 03 00:47:49.016596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.016178 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-credential-socket\") pod \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " Nov 03 00:47:49.016596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.016240 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-kube-api-access-rj6fg\" (UniqueName: \"kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-kube-api-access-rj6fg\") pod \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " Nov 03 00:47:49.016596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.016298 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-data\") pod \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\" (UID: \"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf\") " Nov 03 00:47:49.016596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.016302 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-certs" pod "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" (UID: "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:49.016596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.016347 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-data\") pod \"6dd0c634-3b18-4b15-8438-6cf01db8b352\" (UID: \"6dd0c634-3b18-4b15-8438-6cf01db8b352\") " Nov 03 00:47:49.016891 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.016609 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istiod-ca-cert" pod "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" (UID: "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:49.017339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.017019 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-credential-socket" pod "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" (UID: "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:49.017339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.017067 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-data" pod "6dd0c634-3b18-4b15-8438-6cf01db8b352" (UID: "6dd0c634-3b18-4b15-8438-6cf01db8b352"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:49.017339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.017191 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.017339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.017221 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-socket" pod "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" (UID: "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:49.017339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.017233 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.017339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.017247 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd0c634-3b18-4b15-8438-6cf01db8b352-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.017339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.017261 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.017339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.017275 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.017339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.017290 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.017339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.017301 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-data" pod "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" (UID: "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:49.017339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.017305 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.019103 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.019069 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-token" pod "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" (UID: "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:49.019231 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.019124 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-podinfo" pod "6dd0c634-3b18-4b15-8438-6cf01db8b352" (UID: "6dd0c634-3b18-4b15-8438-6cf01db8b352"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:47:49.019308 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.019287 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-envoy" pod "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" (UID: "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:49.019481 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.019459 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-kube-api-access-zc8k8" pod "6dd0c634-3b18-4b15-8438-6cf01db8b352" (UID: "6dd0c634-3b18-4b15-8438-6cf01db8b352"). InnerVolumeSpecName "kube-api-access-zc8k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:49.019549 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.019501 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-kube-api-access-rj6fg" pod "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" (UID: "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf"). InnerVolumeSpecName "kube-api-access-rj6fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:49.019600 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.019576 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-podinfo" pod "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" (UID: "2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:47:49.020737 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.020718 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-token" pod "6dd0c634-3b18-4b15-8438-6cf01db8b352" (UID: "6dd0c634-3b18-4b15-8438-6cf01db8b352"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:49.021236 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.021219 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-envoy" pod "6dd0c634-3b18-4b15-8438-6cf01db8b352" (UID: "6dd0c634-3b18-4b15-8438-6cf01db8b352"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:49.118419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.118387 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.118419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.118417 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.118672 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.118431 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.118672 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.118441 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8k8\" (UniqueName: \"kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-kube-api-access-zc8k8\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.118672 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.118451 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.118672 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.118459 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.118672 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.118467 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.118672 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.118475 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.118672 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.118484 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj6fg\" (UniqueName: \"kubernetes.io/projected/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-kube-api-access-rj6fg\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.118672 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.118497 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.118672 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.118509 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd0c634-3b18-4b15-8438-6cf01db8b352-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:49.840187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.840149 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" event={"ID":"6dd0c634-3b18-4b15-8438-6cf01db8b352","Type":"ContainerDied","Data":"024dde751a7e22ba9c0a481581bd32ca5f494b09327e1b882a4dbbdd5623ab8d"} Nov 03 00:47:49.840187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.840175 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n" Nov 03 00:47:49.840727 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.840247 6400 scope.go:122] "RemoveContainer" containerID="f019f7cc6e5881b12d8a6d94f1429b84a237c2cfe9fd4a29e30c5ba0c55977f9" Nov 03 00:47:49.841565 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.841534 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" event={"ID":"2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf","Type":"ContainerDied","Data":"879367b800fbd573ee44565c07e7d925f8cd5c8f09368392f09fa6fd15d0add4"} Nov 03 00:47:49.841639 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.841568 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw" Nov 03 00:47:49.850482 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:49.850468 6400 scope.go:122] "RemoveContainer" containerID="6195135f9b7f24909ce27c8c217d0e950927c6d4aac3ddd2af33ae7358e47f7d" Nov 03 00:47:50.003863 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:50.003769 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n"] Nov 03 00:47:50.052666 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:50.052638 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rm28n"] Nov 03 00:47:50.114627 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:50.114598 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw"] Nov 03 00:47:50.124138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:50.124112 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-svljw"] Nov 03 00:47:51.179177 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:51.179145 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" path="/var/lib/kubelet/pods/2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf/volumes" Nov 03 00:47:51.179777 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:51.179755 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd0c634-3b18-4b15-8438-6cf01db8b352" path="/var/lib/kubelet/pods/6dd0c634-3b18-4b15-8438-6cf01db8b352/volumes" Nov 03 00:47:52.964473 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:52.964437 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:54.446025 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:54.445987 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:55.135553 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:55.134144 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:55.429274 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:55.429185 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:57.152506 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:47:57.152469 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:47:57.813764 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:57.813724 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:59.666854 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:59.666812 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:59.857330 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:47:59.857291 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:02.567589 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.567513 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp"] Nov 03 00:48:02.568041 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.567973 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6dd0c634-3b18-4b15-8438-6cf01db8b352" containerName="istio-proxy" Nov 03 00:48:02.568041 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.567985 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd0c634-3b18-4b15-8438-6cf01db8b352" containerName="istio-proxy" Nov 03 00:48:02.568041 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.568003 6400 cpu_manager.go:414] "containerMap: removing container" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerName="istio-proxy" Nov 03 00:48:02.568041 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.568008 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerName="istio-proxy" Nov 03 00:48:02.568170 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.568073 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd0c634-3b18-4b15-8438-6cf01db8b352" containerName="istio-proxy" Nov 03 00:48:02.568170 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.568084 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a52a9c6-c4a0-4fb1-a682-cbc3c279aedf" containerName="istio-proxy" Nov 03 00:48:02.572661 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.572643 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:02.671097 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.671075 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:02.671238 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.671184 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:02.671280 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.671272 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:02.671330 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.671292 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf45h\" (UniqueName: \"kubernetes.io/projected/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-kube-api-access-nf45h\") pod \"maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:02.676983 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.676964 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-c87e2f-serving-cert\"" Nov 03 00:48:02.678966 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.678948 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp"] Nov 03 00:48:02.781592 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.781559 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:02.781747 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.781599 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf45h\" (UniqueName: \"kubernetes.io/projected/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-kube-api-access-nf45h\") pod \"maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:02.781747 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.781646 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:02.781906 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:48:02.781882 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-serving-cert: secret "maas-api-e2e-shared-b-w3-c87e2f-serving-cert" not found Nov 03 00:48:02.782038 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:48:02.781960 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-maas-api-tls podName:a9fc68ce-c4aa-4e6f-b265-8802ee46de54 nodeName:}" failed. No retries permitted until 2026-11-03 00:48:03.281927284 +0000 UTC m=+2530.232867436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-maas-api-tls") pod "maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" (UID: "a9fc68ce-c4aa-4e6f-b265-8802ee46de54") : secret "maas-api-e2e-shared-b-w3-c87e2f-serving-cert" not found Nov 03 00:48:02.782038 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.781978 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:02.782452 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.782435 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:02.784628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.784609 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:02.790604 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.790553 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:02.840098 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.840047 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf45h\" (UniqueName: \"kubernetes.io/projected/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-kube-api-access-nf45h\") pod \"maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:02.964219 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:02.964181 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:03.146151 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:03.146062 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:03.331257 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:03.331220 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:03.350860 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:03.350786 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:03.351427 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:03.351402 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:03.486100 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:03.486024 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:03.859052 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:03.859025 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp"] Nov 03 00:48:03.861148 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:48:03.861122 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fc68ce_c4aa_4e6f_b265_8802ee46de54.slice/crio-4ff71b97f05c10e96878ad2eedd5a3aa083a0b1dc5ac90c402b2f7235fdc2e8d WatchSource:0}: Error finding container 4ff71b97f05c10e96878ad2eedd5a3aa083a0b1dc5ac90c402b2f7235fdc2e8d: Status 404 returned error can't find the container with id 4ff71b97f05c10e96878ad2eedd5a3aa083a0b1dc5ac90c402b2f7235fdc2e8d Nov 03 00:48:03.950000 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:03.949970 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" event={"ID":"a9fc68ce-c4aa-4e6f-b265-8802ee46de54","Type":"ContainerStarted","Data":"4ff71b97f05c10e96878ad2eedd5a3aa083a0b1dc5ac90c402b2f7235fdc2e8d"} Nov 03 00:48:04.965973 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:04.965864 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" event={"ID":"a9fc68ce-c4aa-4e6f-b265-8802ee46de54","Type":"ContainerStarted","Data":"c124c7a27ebccdfcb1b71924de7026f48d5c79c3b8d3f8d97504b85d24472b6a"} Nov 03 00:48:04.965973 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:04.965956 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:05.080371 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:05.080318 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" podStartSLOduration=2.261725342 podStartE2EDuration="3.080304304s" totalImagesPullingTime="818.578962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:05.078761042 +0000 UTC m=+2532.029701215" watchObservedRunningTime="2026-11-03 00:48:05.080304304 +0000 UTC m=+2532.031244459" Nov 03 00:48:07.997323 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:07.997290 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:09.389827 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:09.389786 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:48:10.976361 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:10.976331 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 00:48:12.821412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:12.821375 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:13.021432 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:13.021401 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:17.459554 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:17.459516 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:19.852529 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:19.852493 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92"] Nov 03 00:48:19.856157 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:19.856141 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:19.913546 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:19.913519 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:19.913708 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:19.913557 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:19.913708 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:19.913678 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:19.913847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:19.913791 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:19.913917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:19.913856 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4f5p\" (UniqueName: \"kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-kube-api-access-z4f5p\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:19.914020 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:19.914005 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:19.914073 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:19.914059 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:19.914126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:19.914092 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:19.914191 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:19.914175 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:19.921854 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:19.921804 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ebeeb9-openshift-default-dockercfg-pt5k5\"" Nov 03 00:48:19.960306 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:19.960273 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92"] Nov 03 00:48:20.014868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.014808 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.015070 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.014908 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.015070 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.014934 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.015070 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.014951 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.015234 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.015090 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.015234 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.015130 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.015234 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.015179 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.015386 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.015248 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.015386 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.015286 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4f5p\" (UniqueName: \"kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-kube-api-access-z4f5p\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.015386 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.015311 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.015386 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.015351 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.015585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.015384 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.015585 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.015483 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.015945 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.015926 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.017481 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.017456 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.017576 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.017523 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.071692 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.071669 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.071850 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.071805 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4f5p\" (UniqueName: \"kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-kube-api-access-z4f5p\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.169238 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.169177 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:20.590976 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.590934 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92"] Nov 03 00:48:20.631499 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:48:20.631475 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d69e32_89cc_46ea_b72c_2e73e5bd17c4.slice/crio-758d6d9894404f776bf5337c8a6fc8c28c16de236d75f90f218699cb0523b3b6 WatchSource:0}: Error finding container 758d6d9894404f776bf5337c8a6fc8c28c16de236d75f90f218699cb0523b3b6: Status 404 returned error can't find the container with id 758d6d9894404f776bf5337c8a6fc8c28c16de236d75f90f218699cb0523b3b6 Nov 03 00:48:20.633835 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.633787 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:48:20.633904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.633880 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:48:20.633941 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:20.633916 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:48:21.052988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:21.052950 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" event={"ID":"11d69e32-89cc-46ea-b72c-2e73e5bd17c4","Type":"ContainerStarted","Data":"a271f7385cc68aeb2e100db8c48e8c9b4e148bd999a56c2c1f3d102a647a0a26"} Nov 03 00:48:21.053356 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:21.053012 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" event={"ID":"11d69e32-89cc-46ea-b72c-2e73e5bd17c4","Type":"ContainerStarted","Data":"758d6d9894404f776bf5337c8a6fc8c28c16de236d75f90f218699cb0523b3b6"} Nov 03 00:48:21.168257 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:21.168209 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" podStartSLOduration=2.168195877 podStartE2EDuration="2.168195877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:21.166921633 +0000 UTC m=+2548.117861791" watchObservedRunningTime="2026-11-03 00:48:21.168195877 +0000 UTC m=+2548.119136033" Nov 03 00:48:21.169706 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:21.169681 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:21.171345 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:21.171319 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:48:21.171475 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:21.171381 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" podUID="11d69e32-89cc-46ea-b72c-2e73e5bd17c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:48:22.029216 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:22.029181 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:22.169849 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:22.169798 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:48:22.170197 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:22.169862 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" podUID="11d69e32-89cc-46ea-b72c-2e73e5bd17c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:48:22.202443 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:22.202398 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:22.356297 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:22.356264 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:23.170135 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:23.170100 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:48:23.170574 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:23.170153 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" podUID="11d69e32-89cc-46ea-b72c-2e73e5bd17c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:48:24.170009 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:24.169966 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:48:24.170159 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:24.170028 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" podUID="11d69e32-89cc-46ea-b72c-2e73e5bd17c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:48:25.169884 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:25.169855 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:48:25.170076 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:25.169906 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" podUID="11d69e32-89cc-46ea-b72c-2e73e5bd17c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:48:26.170531 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:26.170501 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:48:26.170933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:26.170561 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" podUID="11d69e32-89cc-46ea-b72c-2e73e5bd17c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:48:26.933760 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:26.933724 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:27.151370 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:48:27.151334 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:48:27.170430 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:27.170405 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 00:48:27.170538 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:27.170462 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" podUID="11d69e32-89cc-46ea-b72c-2e73e5bd17c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 00:48:28.175108 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:28.175078 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:28.175578 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:28.175486 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:28.176171 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:28.176151 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:48:31.667978 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:31.667941 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:36.388391 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:36.388353 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:38.389291 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:38.389251 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:41.657945 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:41.657899 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:41.891844 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:41.888510 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp"] Nov 03 00:48:41.897555 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:41.897520 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:41.925767 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:41.925730 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-tmp-dir\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:41.925954 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:41.925801 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-model-cache\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:41.925954 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:41.925891 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-home\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:41.926057 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:41.926034 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vk9p\" (UniqueName: \"kubernetes.io/projected/226105cb-1776-4571-8e64-f0b0fef9bdd6-kube-api-access-5vk9p\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:41.926108 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:41.926090 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/226105cb-1776-4571-8e64-f0b0fef9bdd6-tls-certs\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:41.926151 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:41.926139 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-dshm\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:41.949801 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:41.949769 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp"] Nov 03 00:48:42.003847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:41.997280 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b2e795\"/\"kube-root-ca.crt\"" Nov 03 00:48:42.003847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:41.997399 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b2e795\"/\"default-dockercfg-vn8vg\"" Nov 03 00:48:42.003847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.003498 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b2e795\"/\"auth-test-model-b2e795-kserve-self-signed-certs\"" Nov 03 00:48:42.003847 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.003614 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b2e795\"/\"openshift-service-ca.crt\"" Nov 03 00:48:42.018326 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.018295 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:42.027445 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.027416 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-model-cache\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:42.027561 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.027454 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-home\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:42.027561 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.027526 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vk9p\" (UniqueName: \"kubernetes.io/projected/226105cb-1776-4571-8e64-f0b0fef9bdd6-kube-api-access-5vk9p\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:42.027561 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.027546 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/226105cb-1776-4571-8e64-f0b0fef9bdd6-tls-certs\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:42.027718 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.027590 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-dshm\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:42.027718 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.027663 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-tmp-dir\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:42.027907 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.027891 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-home\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:42.027982 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.027941 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-model-cache\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:42.028037 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.027994 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-tmp-dir\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:42.030187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.030162 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-dshm\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:42.030278 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.030250 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/226105cb-1776-4571-8e64-f0b0fef9bdd6-tls-certs\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:42.094842 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.094267 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vk9p\" (UniqueName: \"kubernetes.io/projected/226105cb-1776-4571-8e64-f0b0fef9bdd6-kube-api-access-5vk9p\") pod \"auth-test-model-b2e795-kserve-7945f8dc57-cptvp\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:42.218522 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.218429 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:42.634556 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:42.634532 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp"] Nov 03 00:48:42.636133 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:48:42.636102 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod226105cb_1776_4571_8e64_f0b0fef9bdd6.slice/crio-0121b7de0137343f8cc07e4cd2b78153227a17531a9bca4f3f82d523d11da26e WatchSource:0}: Error finding container 0121b7de0137343f8cc07e4cd2b78153227a17531a9bca4f3f82d523d11da26e: Status 404 returned error can't find the container with id 0121b7de0137343f8cc07e4cd2b78153227a17531a9bca4f3f82d523d11da26e Nov 03 00:48:43.209533 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:43.209493 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" event={"ID":"226105cb-1776-4571-8e64-f0b0fef9bdd6","Type":"ContainerStarted","Data":"cde596fa84ace5042e3ae76021da4f6122c4eac97c963ca81103d485d7016802"} Nov 03 00:48:43.209946 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:43.209545 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" event={"ID":"226105cb-1776-4571-8e64-f0b0fef9bdd6","Type":"ContainerStarted","Data":"0121b7de0137343f8cc07e4cd2b78153227a17531a9bca4f3f82d523d11da26e"} Nov 03 00:48:43.219016 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:43.218986 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:43.322889 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:43.322813 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" podStartSLOduration=2.322798676 podStartE2EDuration="2.322798676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:43.322460026 +0000 UTC m=+2570.273400183" watchObservedRunningTime="2026-11-03 00:48:43.322798676 +0000 UTC m=+2570.273738852" Nov 03 00:48:46.500438 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:46.500398 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:49.244573 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:49.244494 6400 generic.go:356] "Generic (PLEG): container finished" podID="728c90f9-2248-493e-adc6-7e51e412154d" containerID="67935ffc9efc96bfe0d96a47b96cc311f1223647cb8edc5bb3f2b40d03a9dca3" exitCode=0 Nov 03 00:48:49.244573 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:49.244557 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" event={"ID":"728c90f9-2248-493e-adc6-7e51e412154d","Type":"ContainerDied","Data":"67935ffc9efc96bfe0d96a47b96cc311f1223647cb8edc5bb3f2b40d03a9dca3"} Nov 03 00:48:49.245044 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:49.244919 6400 scope.go:122] "RemoveContainer" containerID="67935ffc9efc96bfe0d96a47b96cc311f1223647cb8edc5bb3f2b40d03a9dca3" Nov 03 00:48:49.246309 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:49.246281 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:48:49.246393 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:49.246348 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:48:49.246433 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:49.246413 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:48:50.251748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:50.251713 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" event={"ID":"728c90f9-2248-493e-adc6-7e51e412154d","Type":"ContainerStarted","Data":"4873956bad3c1b5220f2183e23dcdb004bd4e97b9c52b756a478ad1f7e9f8132"} Nov 03 00:48:50.418984 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:50.418947 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:48:50.419186 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:50.419119 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:48:50.419186 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:50.419165 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" podUID="728c90f9-2248-493e-adc6-7e51e412154d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 00:48:51.420126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:51.420094 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:48:51.420505 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:51.420144 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" podUID="728c90f9-2248-493e-adc6-7e51e412154d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 00:48:51.558529 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:51.558500 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:51.762704 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:51.762624 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:52.219219 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:52.219144 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:52.231511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:52.231481 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:52.273669 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:52.273642 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:48:52.419569 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:52.419530 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:48:52.419765 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:52.419591 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" podUID="728c90f9-2248-493e-adc6-7e51e412154d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 00:48:53.421355 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:53.421313 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:48:53.421780 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:53.421382 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" podUID="728c90f9-2248-493e-adc6-7e51e412154d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 00:48:54.419719 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:54.419677 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:48:54.419936 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:54.419750 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" podUID="728c90f9-2248-493e-adc6-7e51e412154d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 00:48:55.422927 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:55.422897 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:48:55.423375 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:55.423295 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:48:55.423924 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:55.423903 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 00:48:56.521227 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:48:56.521187 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:57.152144 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:48:57.152111 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:49:01.203381 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:01.203335 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:05.094996 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.094960 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:49:05.645869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.645811 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz"] Nov 03 00:49:05.649007 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.648988 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:05.751831 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.751793 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-maas-api-tls\") pod \"maas-api-e2e-explicit-c39710-84746b4d8-wc2cz\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:05.752004 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.751845 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-metrics-certs\") pod \"maas-api-e2e-explicit-c39710-84746b4d8-wc2cz\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:05.752004 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.751916 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkfvf\" (UniqueName: \"kubernetes.io/projected/fd62c2ed-6525-4328-bd58-2e20030f5e3b-kube-api-access-jkfvf\") pod \"maas-api-e2e-explicit-c39710-84746b4d8-wc2cz\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:05.752095 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.752062 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd62c2ed-6525-4328-bd58-2e20030f5e3b-openshift-service-ca\") pod \"maas-api-e2e-explicit-c39710-84746b4d8-wc2cz\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:05.752181 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.752162 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-c39710-serving-cert\"" Nov 03 00:49:05.754622 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.754601 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz"] Nov 03 00:49:05.853700 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.853665 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd62c2ed-6525-4328-bd58-2e20030f5e3b-openshift-service-ca\") pod \"maas-api-e2e-explicit-c39710-84746b4d8-wc2cz\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:05.853949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.853777 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-maas-api-tls\") pod \"maas-api-e2e-explicit-c39710-84746b4d8-wc2cz\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:05.853949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.853797 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-metrics-certs\") pod \"maas-api-e2e-explicit-c39710-84746b4d8-wc2cz\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:05.853949 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.853856 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfvf\" (UniqueName: \"kubernetes.io/projected/fd62c2ed-6525-4328-bd58-2e20030f5e3b-kube-api-access-jkfvf\") pod \"maas-api-e2e-explicit-c39710-84746b4d8-wc2cz\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:05.854144 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:49:05.853946 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-serving-cert: secret "maas-api-e2e-explicit-c39710-serving-cert" not found Nov 03 00:49:05.854144 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:49:05.854022 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-maas-api-tls podName:fd62c2ed-6525-4328-bd58-2e20030f5e3b nodeName:}" failed. No retries permitted until 2026-11-03 00:49:06.354002692 +0000 UTC m=+2593.304942827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-maas-api-tls") pod "maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" (UID: "fd62c2ed-6525-4328-bd58-2e20030f5e3b") : secret "maas-api-e2e-explicit-c39710-serving-cert" not found Nov 03 00:49:05.854449 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.854425 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd62c2ed-6525-4328-bd58-2e20030f5e3b-openshift-service-ca\") pod \"maas-api-e2e-explicit-c39710-84746b4d8-wc2cz\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:05.856499 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.856477 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-metrics-certs\") pod \"maas-api-e2e-explicit-c39710-84746b4d8-wc2cz\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:05.869879 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.869840 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:05.922460 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:05.922362 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkfvf\" (UniqueName: \"kubernetes.io/projected/fd62c2ed-6525-4328-bd58-2e20030f5e3b-kube-api-access-jkfvf\") pod \"maas-api-e2e-explicit-c39710-84746b4d8-wc2cz\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:06.094270 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:06.094165 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:06.252730 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:06.252654 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:06.364564 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:06.364528 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-maas-api-tls\") pod \"maas-api-e2e-explicit-c39710-84746b4d8-wc2cz\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:06.367218 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:06.367194 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-maas-api-tls\") pod \"maas-api-e2e-explicit-c39710-84746b4d8-wc2cz\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:06.560648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:06.560616 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:06.931078 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:06.931054 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz"] Nov 03 00:49:06.933033 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:49:06.933005 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd62c2ed_6525_4328_bd58_2e20030f5e3b.slice/crio-67e37ab52f60b5381c8295001affc41ec988fe174232aa40c4371888568d0e9a WatchSource:0}: Error finding container 67e37ab52f60b5381c8295001affc41ec988fe174232aa40c4371888568d0e9a: Status 404 returned error can't find the container with id 67e37ab52f60b5381c8295001affc41ec988fe174232aa40c4371888568d0e9a Nov 03 00:49:07.348121 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:07.348089 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" event={"ID":"fd62c2ed-6525-4328-bd58-2e20030f5e3b","Type":"ContainerStarted","Data":"67e37ab52f60b5381c8295001affc41ec988fe174232aa40c4371888568d0e9a"} Nov 03 00:49:08.355060 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:08.355028 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" event={"ID":"fd62c2ed-6525-4328-bd58-2e20030f5e3b","Type":"ContainerStarted","Data":"516c5f980ad754e176fd42f2f3a7d6e23596f27fcb7457b9fd746acd221e3e11"} Nov 03 00:49:08.355448 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:08.355087 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:08.470166 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:08.470116 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" podStartSLOduration=2.587621566 podStartE2EDuration="3.470103011s" totalImagesPullingTime="882.481445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:08.468608328 +0000 UTC m=+2595.419548490" watchObservedRunningTime="2026-11-03 00:49:08.470103011 +0000 UTC m=+2595.421043167" Nov 03 00:49:10.717009 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:10.716969 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:13.195542 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:13.195508 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:14.367131 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:14.367103 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:49:15.357314 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:15.357276 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:20.103846 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:20.103790 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:25.195279 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:25.195241 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:25.791936 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:25.791902 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:27.154566 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:49:27.154530 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:49:28.471773 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:28.471734 6400 generic.go:356] "Generic (PLEG): container finished" podID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerID="b8bbf078a209709ab13ec512e64d21d1263f44d8a9bcad7b8bbae0e5b026411a" exitCode=0 Nov 03 00:49:28.472238 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:28.471780 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" event={"ID":"4df412b7-1ae9-4f96-a92a-85e29370dabb","Type":"ContainerDied","Data":"b8bbf078a209709ab13ec512e64d21d1263f44d8a9bcad7b8bbae0e5b026411a"} Nov 03 00:49:28.472312 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:28.472246 6400 scope.go:122] "RemoveContainer" containerID="b8bbf078a209709ab13ec512e64d21d1263f44d8a9bcad7b8bbae0e5b026411a" Nov 03 00:49:28.473959 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:28.473925 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:49:28.474086 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:28.474011 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:49:28.474086 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:28.474061 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:49:29.480895 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:29.480856 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" event={"ID":"4df412b7-1ae9-4f96-a92a-85e29370dabb","Type":"ContainerStarted","Data":"f0df182a0d526f7f0387a24f941c17ea08b91b8f667572fe81253a29dc5087b3"} Nov 03 00:49:29.875246 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:29.875205 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:49:29.901041 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:29.900996 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:49:29.901263 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:29.901078 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 00:49:30.032188 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:30.032147 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:30.875090 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:30.875049 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:49:30.875576 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:30.875130 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 00:49:31.492844 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:31.488914 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:49:31.875390 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:31.875350 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:49:31.876337 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:31.875408 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 00:49:32.874541 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:32.874510 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:49:32.874738 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:32.874559 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 00:49:33.875273 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:33.875235 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:49:33.875671 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:33.875302 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 00:49:34.834882 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:34.834849 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:34.879316 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:34.879275 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:49:34.879953 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:34.879910 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 00:49:35.879625 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:35.879591 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:49:35.880745 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:35.880723 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 00:49:39.647417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:39.647375 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:40.947834 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:40.947798 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:43.789494 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:43.789462 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:44.612948 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:44.612916 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:48.162450 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:48.162416 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:48.322952 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:48.322921 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:49.319150 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:49.319122 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:53.992266 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:53.992226 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:54.759059 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:54.759026 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:49:57.152867 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:49:57.152800 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:49:58.698402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:49:58.698367 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:00.572148 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:00.572112 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:03.397871 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:03.397786 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:08.120573 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:08.120530 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:12.805364 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:12.805329 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:13.550506 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:13.550475 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:17.526240 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:17.526200 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:22.297165 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:22.297133 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:23.030962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:23.030926 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:27.151586 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:50:27.151551 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:50:27.407807 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:27.407725 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:32.145985 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:32.145467 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:35.917610 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:35.917561 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:36.857538 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:36.857501 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:41.495276 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:41.495234 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:42.180183 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:42.180152 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:50:44.202744 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:44.202708 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:46.276721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:46.276688 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:50.940802 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:50.940758 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:54.542218 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:54.542183 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:55.651228 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:50:55.651191 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:57.152531 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:50:57.152489 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:51:00.391981 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:00.391948 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:02.082209 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:02.082107 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:05.055965 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:05.055929 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:09.813694 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:09.813660 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:13.648451 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:13.648417 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh"] Nov 03 00:51:13.648973 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:13.648674 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" podUID="53b6ff25-e381-44a1-afe6-9ff58381efe5" containerName="istio-proxy" containerID="cri-o://0d1e5f120773375a366af96725423a030316213d33014f5396c6a3127c15c879" gracePeriod=30 Nov 03 00:51:13.948274 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:13.948165 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:14.502779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:14.502745 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:16.413868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:16.413795 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:51:18.907330 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.907303 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:51:18.927596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.927537 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-certs\") pod \"53b6ff25-e381-44a1-afe6-9ff58381efe5\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " Nov 03 00:51:18.927596 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.927568 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-credential-socket\") pod \"53b6ff25-e381-44a1-afe6-9ff58381efe5\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " Nov 03 00:51:18.927748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.927669 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-socket\") pod \"53b6ff25-e381-44a1-afe6-9ff58381efe5\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " Nov 03 00:51:18.927748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.927706 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-podinfo\") pod \"53b6ff25-e381-44a1-afe6-9ff58381efe5\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " Nov 03 00:51:18.927748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.927733 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b6ff25-e381-44a1-afe6-9ff58381efe5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b6ff25-e381-44a1-afe6-9ff58381efe5-istiod-ca-cert\") pod \"53b6ff25-e381-44a1-afe6-9ff58381efe5\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " Nov 03 00:51:18.927944 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.927754 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-kube-api-access-hwpcg\" (UniqueName: \"kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-kube-api-access-hwpcg\") pod \"53b6ff25-e381-44a1-afe6-9ff58381efe5\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " Nov 03 00:51:18.927944 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.927765 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-certs" pod "53b6ff25-e381-44a1-afe6-9ff58381efe5" (UID: "53b6ff25-e381-44a1-afe6-9ff58381efe5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:51:18.927944 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.927832 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-token\" (UniqueName: \"kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-token\") pod \"53b6ff25-e381-44a1-afe6-9ff58381efe5\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " Nov 03 00:51:18.927944 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.927879 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-data\") pod \"53b6ff25-e381-44a1-afe6-9ff58381efe5\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " Nov 03 00:51:18.927944 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.927874 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-credential-socket" pod "53b6ff25-e381-44a1-afe6-9ff58381efe5" (UID: "53b6ff25-e381-44a1-afe6-9ff58381efe5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:51:18.927944 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.927913 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-envoy\") pod \"53b6ff25-e381-44a1-afe6-9ff58381efe5\" (UID: \"53b6ff25-e381-44a1-afe6-9ff58381efe5\") " Nov 03 00:51:18.928223 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.927953 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-socket" pod "53b6ff25-e381-44a1-afe6-9ff58381efe5" (UID: "53b6ff25-e381-44a1-afe6-9ff58381efe5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:51:18.928223 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.928161 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b6ff25-e381-44a1-afe6-9ff58381efe5-istiod-ca-cert" pod "53b6ff25-e381-44a1-afe6-9ff58381efe5" (UID: "53b6ff25-e381-44a1-afe6-9ff58381efe5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:51:18.928363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.928344 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-data" pod "53b6ff25-e381-44a1-afe6-9ff58381efe5" (UID: "53b6ff25-e381-44a1-afe6-9ff58381efe5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:51:18.928449 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.928434 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:18.928671 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.928469 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b6ff25-e381-44a1-afe6-9ff58381efe5-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:18.928671 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.928478 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:18.928671 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.928488 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:18.930188 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.930165 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-podinfo" pod "53b6ff25-e381-44a1-afe6-9ff58381efe5" (UID: "53b6ff25-e381-44a1-afe6-9ff58381efe5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:51:18.930285 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.930219 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-envoy" pod "53b6ff25-e381-44a1-afe6-9ff58381efe5" (UID: "53b6ff25-e381-44a1-afe6-9ff58381efe5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:51:18.930504 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.930485 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-kube-api-access-hwpcg" pod "53b6ff25-e381-44a1-afe6-9ff58381efe5" (UID: "53b6ff25-e381-44a1-afe6-9ff58381efe5"). InnerVolumeSpecName "kube-api-access-hwpcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:51:18.930553 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:18.930521 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-token" pod "53b6ff25-e381-44a1-afe6-9ff58381efe5" (UID: "53b6ff25-e381-44a1-afe6-9ff58381efe5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:51:19.030126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.030095 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:19.030126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.030121 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:19.030126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.030130 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:19.030370 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.030139 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b6ff25-e381-44a1-afe6-9ff58381efe5-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:19.030370 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.030148 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwpcg\" (UniqueName: \"kubernetes.io/projected/53b6ff25-e381-44a1-afe6-9ff58381efe5-kube-api-access-hwpcg\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:51:19.132970 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.132928 6400 generic.go:356] "Generic (PLEG): container finished" podID="53b6ff25-e381-44a1-afe6-9ff58381efe5" containerID="0d1e5f120773375a366af96725423a030316213d33014f5396c6a3127c15c879" exitCode=0 Nov 03 00:51:19.133136 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.132976 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" Nov 03 00:51:19.133136 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.132977 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" event={"ID":"53b6ff25-e381-44a1-afe6-9ff58381efe5","Type":"ContainerDied","Data":"0d1e5f120773375a366af96725423a030316213d33014f5396c6a3127c15c879"} Nov 03 00:51:19.133136 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.133103 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh" event={"ID":"53b6ff25-e381-44a1-afe6-9ff58381efe5","Type":"ContainerDied","Data":"07b8c558fd18099407a134a2227d976bd0ee7829f70024540ca35524e88fbaed"} Nov 03 00:51:19.133136 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.133127 6400 scope.go:122] "RemoveContainer" containerID="0d1e5f120773375a366af96725423a030316213d33014f5396c6a3127c15c879" Nov 03 00:51:19.144517 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.144494 6400 scope.go:122] "RemoveContainer" containerID="0d1e5f120773375a366af96725423a030316213d33014f5396c6a3127c15c879" Nov 03 00:51:19.144865 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:51:19.144843 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1e5f120773375a366af96725423a030316213d33014f5396c6a3127c15c879\": container with ID starting with 0d1e5f120773375a366af96725423a030316213d33014f5396c6a3127c15c879 not found: ID does not exist" containerID="0d1e5f120773375a366af96725423a030316213d33014f5396c6a3127c15c879" Nov 03 00:51:19.144959 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.144877 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1e5f120773375a366af96725423a030316213d33014f5396c6a3127c15c879"} err="failed to get container status \"0d1e5f120773375a366af96725423a030316213d33014f5396c6a3127c15c879\": rpc error: code = NotFound desc = could not find container \"0d1e5f120773375a366af96725423a030316213d33014f5396c6a3127c15c879\": container with ID starting with 0d1e5f120773375a366af96725423a030316213d33014f5396c6a3127c15c879 not found: ID does not exist" Nov 03 00:51:19.221113 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.221039 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:19.297220 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.297184 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh"] Nov 03 00:51:19.300309 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:19.300286 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-wpvdh"] Nov 03 00:51:21.179610 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:21.179574 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b6ff25-e381-44a1-afe6-9ff58381efe5" path="/var/lib/kubelet/pods/53b6ff25-e381-44a1-afe6-9ff58381efe5/volumes" Nov 03 00:51:24.013360 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:24.013324 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:27.153394 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:51:27.153356 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:51:28.664469 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:28.664429 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:33.400019 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:33.399980 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:38.129791 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:38.129750 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:41.902967 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:41.902929 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:42.817545 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:42.817509 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:46.934785 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:46.934748 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:47.617180 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:47.617136 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:50.585713 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:50.585668 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:52.282006 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:52.281969 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:57.052928 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:51:57.052891 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:57.153446 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:51:57.153413 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:52:01.708811 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:01.708774 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:06.433223 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:06.433187 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:09.422227 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:09.422191 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:52:11.100098 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:11.100064 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:12.294436 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:12.294405 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:14.234019 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:14.233987 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:15.028183 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:15.028150 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:15.937624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:15.937583 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:20.629527 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:20.629491 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:24.023368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:24.023326 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:24.905227 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:24.905189 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:25.077461 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:25.077418 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:25.486403 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:25.486367 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:27.152812 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:52:27.152769 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:52:29.816614 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:29.816577 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:30.368607 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:30.368570 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:31.463440 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:31.463358 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:35.103332 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:35.103291 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:39.055425 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.055392 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz"] Nov 03 00:52:39.055899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.055593 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" podUID="fd62c2ed-6525-4328-bd58-2e20030f5e3b" containerName="maas-api" containerID="cri-o://516c5f980ad754e176fd42f2f3a7d6e23596f27fcb7457b9fd746acd221e3e11" gracePeriod=30 Nov 03 00:52:39.189285 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.189239 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:39.375961 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.375931 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:52:39.425326 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.425285 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-maas-api-tls\") pod \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " Nov 03 00:52:39.425515 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.425430 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd62c2ed-6525-4328-bd58-2e20030f5e3b-kube-api-access-jkfvf\" (UniqueName: \"kubernetes.io/projected/fd62c2ed-6525-4328-bd58-2e20030f5e3b-kube-api-access-jkfvf\") pod \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " Nov 03 00:52:39.425515 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.425475 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-metrics-certs\") pod \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " Nov 03 00:52:39.425515 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.425510 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd62c2ed-6525-4328-bd58-2e20030f5e3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd62c2ed-6525-4328-bd58-2e20030f5e3b-openshift-service-ca\") pod \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\" (UID: \"fd62c2ed-6525-4328-bd58-2e20030f5e3b\") " Nov 03 00:52:39.425945 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.425916 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd62c2ed-6525-4328-bd58-2e20030f5e3b-openshift-service-ca" pod "fd62c2ed-6525-4328-bd58-2e20030f5e3b" (UID: "fd62c2ed-6525-4328-bd58-2e20030f5e3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:52:39.426167 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.426014 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd62c2ed-6525-4328-bd58-2e20030f5e3b-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:39.427801 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.427777 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-maas-api-tls" pod "fd62c2ed-6525-4328-bd58-2e20030f5e3b" (UID: "fd62c2ed-6525-4328-bd58-2e20030f5e3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:52:39.427942 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.427796 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd62c2ed-6525-4328-bd58-2e20030f5e3b-kube-api-access-jkfvf" pod "fd62c2ed-6525-4328-bd58-2e20030f5e3b" (UID: "fd62c2ed-6525-4328-bd58-2e20030f5e3b"). InnerVolumeSpecName "kube-api-access-jkfvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:39.427942 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.427849 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-metrics-certs" pod "fd62c2ed-6525-4328-bd58-2e20030f5e3b" (UID: "fd62c2ed-6525-4328-bd58-2e20030f5e3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:52:39.527771 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.527722 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkfvf\" (UniqueName: \"kubernetes.io/projected/fd62c2ed-6525-4328-bd58-2e20030f5e3b-kube-api-access-jkfvf\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:39.527771 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.527762 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:39.527771 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.527778 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd62c2ed-6525-4328-bd58-2e20030f5e3b-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:39.638633 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.638531 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" Nov 03 00:52:39.638923 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.638560 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" event={"ID":"fd62c2ed-6525-4328-bd58-2e20030f5e3b","Type":"ContainerDied","Data":"516c5f980ad754e176fd42f2f3a7d6e23596f27fcb7457b9fd746acd221e3e11"} Nov 03 00:52:39.639026 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.639002 6400 scope.go:122] "RemoveContainer" containerID="516c5f980ad754e176fd42f2f3a7d6e23596f27fcb7457b9fd746acd221e3e11" Nov 03 00:52:39.639453 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.639427 6400 generic.go:356] "Generic (PLEG): container finished" podID="fd62c2ed-6525-4328-bd58-2e20030f5e3b" containerID="516c5f980ad754e176fd42f2f3a7d6e23596f27fcb7457b9fd746acd221e3e11" exitCode=0 Nov 03 00:52:39.640929 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.640908 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" event={"ID":"fd62c2ed-6525-4328-bd58-2e20030f5e3b","Type":"ContainerDied","Data":"67e37ab52f60b5381c8295001affc41ec988fe174232aa40c4371888568d0e9a"} Nov 03 00:52:39.652647 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.652626 6400 scope.go:122] "RemoveContainer" containerID="516c5f980ad754e176fd42f2f3a7d6e23596f27fcb7457b9fd746acd221e3e11" Nov 03 00:52:39.653016 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:52:39.652993 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516c5f980ad754e176fd42f2f3a7d6e23596f27fcb7457b9fd746acd221e3e11\": container with ID starting with 516c5f980ad754e176fd42f2f3a7d6e23596f27fcb7457b9fd746acd221e3e11 not found: ID does not exist" containerID="516c5f980ad754e176fd42f2f3a7d6e23596f27fcb7457b9fd746acd221e3e11" Nov 03 00:52:39.653096 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.653026 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516c5f980ad754e176fd42f2f3a7d6e23596f27fcb7457b9fd746acd221e3e11"} err="failed to get container status \"516c5f980ad754e176fd42f2f3a7d6e23596f27fcb7457b9fd746acd221e3e11\": rpc error: code = NotFound desc = could not find container \"516c5f980ad754e176fd42f2f3a7d6e23596f27fcb7457b9fd746acd221e3e11\": container with ID starting with 516c5f980ad754e176fd42f2f3a7d6e23596f27fcb7457b9fd746acd221e3e11 not found: ID does not exist" Nov 03 00:52:39.804346 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.804311 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz"] Nov 03 00:52:39.807873 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:39.807850 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz"] Nov 03 00:52:40.923624 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:40.923589 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:41.179288 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:41.179210 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd62c2ed-6525-4328-bd58-2e20030f5e3b" path="/var/lib/kubelet/pods/fd62c2ed-6525-4328-bd58-2e20030f5e3b/volumes" Nov 03 00:52:42.363731 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:42.363686 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c39710-84746b4d8-wc2cz" podUID="fd62c2ed-6525-4328-bd58-2e20030f5e3b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.110:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:52:44.454384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:44.454343 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:44.758875 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:44.758770 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:45.755358 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:45.755309 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:46.085324 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:46.085290 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:46.570293 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:46.570258 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:46.733988 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:46.733795 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:47.349851 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:47.349806 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:47.969894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:47.968234 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:48.218270 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:48.218224 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:49.672213 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:49.672178 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:49.885352 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:49.885320 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:50.055672 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:50.055471 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:50.912336 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:50.912298 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:53.718516 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:53.718471 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:52:55.657917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:52:55.656441 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:57.152886 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:52:57.152848 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:53:00.311082 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:00.311044 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:05.098202 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:05.098156 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:08.869886 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:08.869846 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:09.675012 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:09.674977 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:14.322291 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:14.322251 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:14.582856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:14.582744 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:18.973365 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:18.973332 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:23.745404 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:23.745358 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:27.152600 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:53:27.152565 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:53:28.376444 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:28.376409 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:28.666961 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:28.666873 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9"] Nov 03 00:53:28.667129 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:28.667083 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" podUID="f5f8a22b-db08-4df6-bfd6-e54789ac29e2" containerName="istio-proxy" containerID="cri-o://46d04b65d4c5fa88d1bfe027f19475563dfa56c519fcfec2d7abe141c7f9f78d" gracePeriod=30 Nov 03 00:53:29.019971 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:29.019893 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:53:33.096551 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:33.096513 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:33.931866 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:33.931775 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:53:33.949147 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:33.949115 6400 generic.go:356] "Generic (PLEG): container finished" podID="f5f8a22b-db08-4df6-bfd6-e54789ac29e2" containerID="46d04b65d4c5fa88d1bfe027f19475563dfa56c519fcfec2d7abe141c7f9f78d" exitCode=0 Nov 03 00:53:33.949322 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:33.949169 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" Nov 03 00:53:33.949322 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:33.949189 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" event={"ID":"f5f8a22b-db08-4df6-bfd6-e54789ac29e2","Type":"ContainerDied","Data":"46d04b65d4c5fa88d1bfe027f19475563dfa56c519fcfec2d7abe141c7f9f78d"} Nov 03 00:53:33.949322 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:33.949250 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9" event={"ID":"f5f8a22b-db08-4df6-bfd6-e54789ac29e2","Type":"ContainerDied","Data":"ba612dab0a07b63bb23f95957af8470205327f73fa2830ff50df1418ad454365"} Nov 03 00:53:33.949322 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:33.949277 6400 scope.go:122] "RemoveContainer" containerID="46d04b65d4c5fa88d1bfe027f19475563dfa56c519fcfec2d7abe141c7f9f78d" Nov 03 00:53:33.959067 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:33.959042 6400 scope.go:122] "RemoveContainer" containerID="46d04b65d4c5fa88d1bfe027f19475563dfa56c519fcfec2d7abe141c7f9f78d" Nov 03 00:53:33.959412 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:53:33.959386 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d04b65d4c5fa88d1bfe027f19475563dfa56c519fcfec2d7abe141c7f9f78d\": container with ID starting with 46d04b65d4c5fa88d1bfe027f19475563dfa56c519fcfec2d7abe141c7f9f78d not found: ID does not exist" containerID="46d04b65d4c5fa88d1bfe027f19475563dfa56c519fcfec2d7abe141c7f9f78d" Nov 03 00:53:33.959492 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:33.959422 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d04b65d4c5fa88d1bfe027f19475563dfa56c519fcfec2d7abe141c7f9f78d"} err="failed to get container status \"46d04b65d4c5fa88d1bfe027f19475563dfa56c519fcfec2d7abe141c7f9f78d\": rpc error: code = NotFound desc = could not find container \"46d04b65d4c5fa88d1bfe027f19475563dfa56c519fcfec2d7abe141c7f9f78d\": container with ID starting with 46d04b65d4c5fa88d1bfe027f19475563dfa56c519fcfec2d7abe141c7f9f78d not found: ID does not exist" Nov 03 00:53:34.041232 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041204 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-data\") pod \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " Nov 03 00:53:34.041422 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041258 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-certs\") pod \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " Nov 03 00:53:34.041422 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041290 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-podinfo\") pod \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " Nov 03 00:53:34.041422 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041334 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-kube-api-access-874zd\" (UniqueName: \"kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-kube-api-access-874zd\") pod \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " Nov 03 00:53:34.041422 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041387 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-socket\") pod \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " Nov 03 00:53:34.041422 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041415 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istiod-ca-cert\") pod \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " Nov 03 00:53:34.041697 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041520 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-credential-socket\") pod \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " Nov 03 00:53:34.041697 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041539 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-data" pod "f5f8a22b-db08-4df6-bfd6-e54789ac29e2" (UID: "f5f8a22b-db08-4df6-bfd6-e54789ac29e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.041697 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041559 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-certs" pod "f5f8a22b-db08-4df6-bfd6-e54789ac29e2" (UID: "f5f8a22b-db08-4df6-bfd6-e54789ac29e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.041697 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041584 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-token\" (UniqueName: \"kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-token\") pod \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " Nov 03 00:53:34.041697 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041626 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-envoy\") pod \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\" (UID: \"f5f8a22b-db08-4df6-bfd6-e54789ac29e2\") " Nov 03 00:53:34.041979 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041779 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-credential-socket" pod "f5f8a22b-db08-4df6-bfd6-e54789ac29e2" (UID: "f5f8a22b-db08-4df6-bfd6-e54789ac29e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.041979 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041844 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-socket" pod "f5f8a22b-db08-4df6-bfd6-e54789ac29e2" (UID: "f5f8a22b-db08-4df6-bfd6-e54789ac29e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.041979 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.041908 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istiod-ca-cert" pod "f5f8a22b-db08-4df6-bfd6-e54789ac29e2" (UID: "f5f8a22b-db08-4df6-bfd6-e54789ac29e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:53:34.042266 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.042181 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.042266 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.042207 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.042266 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.042220 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.042266 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.042233 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.042266 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.042247 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.043752 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.043731 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-podinfo" pod "f5f8a22b-db08-4df6-bfd6-e54789ac29e2" (UID: "f5f8a22b-db08-4df6-bfd6-e54789ac29e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:53:34.044273 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.044256 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-token" pod "f5f8a22b-db08-4df6-bfd6-e54789ac29e2" (UID: "f5f8a22b-db08-4df6-bfd6-e54789ac29e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:34.044413 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.044392 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-kube-api-access-874zd" pod "f5f8a22b-db08-4df6-bfd6-e54789ac29e2" (UID: "f5f8a22b-db08-4df6-bfd6-e54789ac29e2"). InnerVolumeSpecName "kube-api-access-874zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:34.044475 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.044392 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-envoy" pod "f5f8a22b-db08-4df6-bfd6-e54789ac29e2" (UID: "f5f8a22b-db08-4df6-bfd6-e54789ac29e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.143438 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.143398 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.143438 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.143433 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.143438 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.143443 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.143930 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.143453 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-874zd\" (UniqueName: \"kubernetes.io/projected/f5f8a22b-db08-4df6-bfd6-e54789ac29e2-kube-api-access-874zd\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.368675 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.368642 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9"] Nov 03 00:53:34.374068 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:34.374036 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-s49g9"] Nov 03 00:53:35.178977 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:35.178947 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f8a22b-db08-4df6-bfd6-e54789ac29e2" path="/var/lib/kubelet/pods/f5f8a22b-db08-4df6-bfd6-e54789ac29e2/volumes" Nov 03 00:53:36.208917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.208885 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw"] Nov 03 00:53:36.209721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.209492 6400 cpu_manager.go:414] "containerMap: removing container" podUID="fd62c2ed-6525-4328-bd58-2e20030f5e3b" containerName="maas-api" Nov 03 00:53:36.209721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.209511 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd62c2ed-6525-4328-bd58-2e20030f5e3b" containerName="maas-api" Nov 03 00:53:36.209721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.209522 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f5f8a22b-db08-4df6-bfd6-e54789ac29e2" containerName="istio-proxy" Nov 03 00:53:36.209721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.209527 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f8a22b-db08-4df6-bfd6-e54789ac29e2" containerName="istio-proxy" Nov 03 00:53:36.209721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.209547 6400 cpu_manager.go:414] "containerMap: removing container" podUID="53b6ff25-e381-44a1-afe6-9ff58381efe5" containerName="istio-proxy" Nov 03 00:53:36.209721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.209552 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b6ff25-e381-44a1-afe6-9ff58381efe5" containerName="istio-proxy" Nov 03 00:53:36.209721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.209659 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd62c2ed-6525-4328-bd58-2e20030f5e3b" containerName="maas-api" Nov 03 00:53:36.209721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.209668 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f8a22b-db08-4df6-bfd6-e54789ac29e2" containerName="istio-proxy" Nov 03 00:53:36.209721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.209679 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b6ff25-e381-44a1-afe6-9ff58381efe5" containerName="istio-proxy" Nov 03 00:53:36.212726 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.212710 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.315498 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.315457 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-1f20dd\"/\"openshift-service-ca.crt\"" Nov 03 00:53:36.315498 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.315482 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-1f20dd\"/\"default-dockercfg-nfcsr\"" Nov 03 00:53:36.315498 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.315493 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-1f20dd\"/\"kube-root-ca.crt\"" Nov 03 00:53:36.315856 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.315689 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-1f20dd\"/\"noten-model-1f20dd-kserve-self-signed-certs\"" Nov 03 00:53:36.317581 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.317562 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw"] Nov 03 00:53:36.333740 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.333714 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:36.372109 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.371152 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29ksr\" (UniqueName: \"kubernetes.io/projected/82c39d12-1304-4ced-a9ad-9163d91d6cbe-kube-api-access-29ksr\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.372109 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.371204 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-home\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.372109 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.371243 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-model-cache\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.372109 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.371278 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-dshm\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.372109 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.371427 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tls-certs\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.372109 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.371745 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tmp-dir\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.473215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.473132 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29ksr\" (UniqueName: \"kubernetes.io/projected/82c39d12-1304-4ced-a9ad-9163d91d6cbe-kube-api-access-29ksr\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.473215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.473168 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-home\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.473215 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.473193 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-model-cache\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.473452 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.473383 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-dshm\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.473452 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.473447 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tls-certs\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.473583 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.473554 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-home\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.473646 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.473589 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tmp-dir\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.473646 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.473597 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-model-cache\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.473894 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.473816 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tmp-dir\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.476442 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.476417 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-dshm\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.476661 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.476639 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tls-certs\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.532338 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.532307 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29ksr\" (UniqueName: \"kubernetes.io/projected/82c39d12-1304-4ced-a9ad-9163d91d6cbe-kube-api-access-29ksr\") pod \"noten-model-1f20dd-kserve-84d8944c65-tkrbw\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:36.824100 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:36.824065 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:37.205100 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:53:37.205066 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82c39d12_1304_4ced_a9ad_9163d91d6cbe.slice/crio-b0c04342bbe14c1fa3b40b65f3e4f2febb251ccefe1ba34026c62273fb92d2dc WatchSource:0}: Error finding container b0c04342bbe14c1fa3b40b65f3e4f2febb251ccefe1ba34026c62273fb92d2dc: Status 404 returned error can't find the container with id b0c04342bbe14c1fa3b40b65f3e4f2febb251ccefe1ba34026c62273fb92d2dc Nov 03 00:53:37.205529 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:37.205502 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw"] Nov 03 00:53:37.813187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:37.813129 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:37.974333 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:37.974299 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" event={"ID":"82c39d12-1304-4ced-a9ad-9163d91d6cbe","Type":"ContainerStarted","Data":"b1722565b8ec72cba1d56a17bc86da07b35f04e1d3c21730e1fc68ab4ac2afdb"} Nov 03 00:53:37.974333 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:37.974341 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" event={"ID":"82c39d12-1304-4ced-a9ad-9163d91d6cbe","Type":"ContainerStarted","Data":"b0c04342bbe14c1fa3b40b65f3e4f2febb251ccefe1ba34026c62273fb92d2dc"} Nov 03 00:53:38.089380 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:38.089278 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" podStartSLOduration=2.089262701 podStartE2EDuration="2.089262701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:38.087648493 +0000 UTC m=+2865.038588651" watchObservedRunningTime="2026-11-03 00:53:38.089262701 +0000 UTC m=+2865.040202858" Nov 03 00:53:38.825150 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:38.825112 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:42.503708 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:42.503670 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:46.824758 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:46.824720 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:46.837032 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:46.837007 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:47.035604 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:47.035578 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:53:47.209477 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:47.209395 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:51.901565 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:51.901529 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:56.617064 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:53:56.617029 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:57.152932 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:53:57.152883 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:54:01.309766 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:01.309670 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:04.777610 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:04.777575 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:04.945260 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:04.945217 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:06.003904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:06.003863 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:07.373085 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:07.373052 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh"] Nov 03 00:54:07.373533 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:07.373245 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" podUID="3e6b2116-c767-4c94-867f-448307c7bb4f" containerName="istio-proxy" containerID="cri-o://81fcf90a30ff68df4c9c736c87db8aab9504c54c5e147b737a9624d79469df03" gracePeriod=30 Nov 03 00:54:07.484995 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:07.484956 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92"] Nov 03 00:54:07.485229 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:07.485206 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" podUID="11d69e32-89cc-46ea-b72c-2e73e5bd17c4" containerName="istio-proxy" containerID="cri-o://a271f7385cc68aeb2e100db8c48e8c9b4e148bd999a56c2c1f3d102a647a0a26" gracePeriod=30 Nov 03 00:54:10.586583 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:10.586543 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:54:10.587011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:10.586618 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" podUID="3e6b2116-c767-4c94-867f-448307c7bb4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:54:10.793582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:10.793542 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:12.640811 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.640741 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:54:12.689120 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.689089 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-podinfo\") pod \"3e6b2116-c767-4c94-867f-448307c7bb4f\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " Nov 03 00:54:12.689295 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.689203 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-credential-socket\") pod \"3e6b2116-c767-4c94-867f-448307c7bb4f\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " Nov 03 00:54:12.689367 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.689291 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-kube-api-access-tlbzd\" (UniqueName: \"kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-kube-api-access-tlbzd\") pod \"3e6b2116-c767-4c94-867f-448307c7bb4f\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " Nov 03 00:54:12.689420 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.689360 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-envoy\") pod \"3e6b2116-c767-4c94-867f-448307c7bb4f\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " Nov 03 00:54:12.689420 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.689390 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-socket\") pod \"3e6b2116-c767-4c94-867f-448307c7bb4f\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " Nov 03 00:54:12.689516 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.689448 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-data\") pod \"3e6b2116-c767-4c94-867f-448307c7bb4f\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " Nov 03 00:54:12.689516 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.689484 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-credential-socket" pod "3e6b2116-c767-4c94-867f-448307c7bb4f" (UID: "3e6b2116-c767-4c94-867f-448307c7bb4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:12.689617 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.689524 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-token\" (UniqueName: \"kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-token\") pod \"3e6b2116-c767-4c94-867f-448307c7bb4f\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " Nov 03 00:54:12.689617 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.689554 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-certs\") pod \"3e6b2116-c767-4c94-867f-448307c7bb4f\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " Nov 03 00:54:12.689617 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.689589 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6b2116-c767-4c94-867f-448307c7bb4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6b2116-c767-4c94-867f-448307c7bb4f-istiod-ca-cert\") pod \"3e6b2116-c767-4c94-867f-448307c7bb4f\" (UID: \"3e6b2116-c767-4c94-867f-448307c7bb4f\") " Nov 03 00:54:12.689995 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.689678 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-socket" pod "3e6b2116-c767-4c94-867f-448307c7bb4f" (UID: "3e6b2116-c767-4c94-867f-448307c7bb4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:12.690089 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.689881 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-certs" pod "3e6b2116-c767-4c94-867f-448307c7bb4f" (UID: "3e6b2116-c767-4c94-867f-448307c7bb4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:12.690384 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.690333 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-data" pod "3e6b2116-c767-4c94-867f-448307c7bb4f" (UID: "3e6b2116-c767-4c94-867f-448307c7bb4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:12.690487 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.690459 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6b2116-c767-4c94-867f-448307c7bb4f-istiod-ca-cert" pod "3e6b2116-c767-4c94-867f-448307c7bb4f" (UID: "3e6b2116-c767-4c94-867f-448307c7bb4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:12.690662 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.690640 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.690746 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.690666 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.690746 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.690682 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.690746 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.690695 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.690746 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.690708 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6b2116-c767-4c94-867f-448307c7bb4f-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.692121 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.692087 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-podinfo" pod "3e6b2116-c767-4c94-867f-448307c7bb4f" (UID: "3e6b2116-c767-4c94-867f-448307c7bb4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:54:12.692492 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.692471 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-token" pod "3e6b2116-c767-4c94-867f-448307c7bb4f" (UID: "3e6b2116-c767-4c94-867f-448307c7bb4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:12.692582 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.692531 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-envoy" pod "3e6b2116-c767-4c94-867f-448307c7bb4f" (UID: "3e6b2116-c767-4c94-867f-448307c7bb4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:12.693132 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.693109 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-kube-api-access-tlbzd" pod "3e6b2116-c767-4c94-867f-448307c7bb4f" (UID: "3e6b2116-c767-4c94-867f-448307c7bb4f"). InnerVolumeSpecName "kube-api-access-tlbzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:12.745572 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.745545 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:54:12.791923 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.791886 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-podinfo\") pod \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " Nov 03 00:54:12.792126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.791963 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-certs\") pod \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " Nov 03 00:54:12.792126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.792055 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-token\" (UniqueName: \"kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-token\") pod \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " Nov 03 00:54:12.792126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.792086 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-credential-socket\") pod \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " Nov 03 00:54:12.792300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.792134 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-envoy\") pod \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " Nov 03 00:54:12.792300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.792189 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istiod-ca-cert\") pod \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " Nov 03 00:54:12.792300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.792231 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-data\") pod \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " Nov 03 00:54:12.792300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.792270 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-kube-api-access-z4f5p\" (UniqueName: \"kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-kube-api-access-z4f5p\") pod \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " Nov 03 00:54:12.792300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.792285 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-certs" pod "11d69e32-89cc-46ea-b72c-2e73e5bd17c4" (UID: "11d69e32-89cc-46ea-b72c-2e73e5bd17c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:12.792525 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.792314 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-socket\") pod \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\" (UID: \"11d69e32-89cc-46ea-b72c-2e73e5bd17c4\") " Nov 03 00:54:12.792525 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.792386 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-credential-socket" pod "11d69e32-89cc-46ea-b72c-2e73e5bd17c4" (UID: "11d69e32-89cc-46ea-b72c-2e73e5bd17c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:12.792525 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.792498 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-data" pod "11d69e32-89cc-46ea-b72c-2e73e5bd17c4" (UID: "11d69e32-89cc-46ea-b72c-2e73e5bd17c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:12.792814 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.792769 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-socket" pod "11d69e32-89cc-46ea-b72c-2e73e5bd17c4" (UID: "11d69e32-89cc-46ea-b72c-2e73e5bd17c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:12.793010 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.792992 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istiod-ca-cert" pod "11d69e32-89cc-46ea-b72c-2e73e5bd17c4" (UID: "11d69e32-89cc-46ea-b72c-2e73e5bd17c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:12.793070 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.793014 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.793070 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.793039 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.793070 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.793056 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlbzd\" (UniqueName: \"kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-kube-api-access-tlbzd\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.793233 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.793073 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.793233 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.793088 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.793233 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.793102 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e6b2116-c767-4c94-867f-448307c7bb4f-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.793233 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.793115 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.793233 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.793130 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.794357 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.794333 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-token" pod "11d69e32-89cc-46ea-b72c-2e73e5bd17c4" (UID: "11d69e32-89cc-46ea-b72c-2e73e5bd17c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:12.794637 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.794616 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-podinfo" pod "11d69e32-89cc-46ea-b72c-2e73e5bd17c4" (UID: "11d69e32-89cc-46ea-b72c-2e73e5bd17c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:54:12.794899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.794875 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-envoy" pod "11d69e32-89cc-46ea-b72c-2e73e5bd17c4" (UID: "11d69e32-89cc-46ea-b72c-2e73e5bd17c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:12.794982 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.794917 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-kube-api-access-z4f5p" pod "11d69e32-89cc-46ea-b72c-2e73e5bd17c4" (UID: "11d69e32-89cc-46ea-b72c-2e73e5bd17c4"). InnerVolumeSpecName "kube-api-access-z4f5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:12.876549 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.876511 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw"] Nov 03 00:54:12.894084 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.894000 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.894084 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.894030 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.894084 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.894044 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4f5p\" (UniqueName: \"kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-kube-api-access-z4f5p\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.894084 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.894057 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.894084 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.894068 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d69e32-89cc-46ea-b72c-2e73e5bd17c4-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:12.937558 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:12.937523 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:13.108331 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.108296 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:13.202383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.202271 6400 generic.go:356] "Generic (PLEG): container finished" podID="11d69e32-89cc-46ea-b72c-2e73e5bd17c4" containerID="a271f7385cc68aeb2e100db8c48e8c9b4e148bd999a56c2c1f3d102a647a0a26" exitCode=0 Nov 03 00:54:13.202814 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.202769 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" event={"ID":"11d69e32-89cc-46ea-b72c-2e73e5bd17c4","Type":"ContainerDied","Data":"a271f7385cc68aeb2e100db8c48e8c9b4e148bd999a56c2c1f3d102a647a0a26"} Nov 03 00:54:13.203110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.203076 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" event={"ID":"11d69e32-89cc-46ea-b72c-2e73e5bd17c4","Type":"ContainerDied","Data":"758d6d9894404f776bf5337c8a6fc8c28c16de236d75f90f218699cb0523b3b6"} Nov 03 00:54:13.203267 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.203253 6400 scope.go:122] "RemoveContainer" containerID="a271f7385cc68aeb2e100db8c48e8c9b4e148bd999a56c2c1f3d102a647a0a26" Nov 03 00:54:13.203560 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.203545 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92" Nov 03 00:54:13.219484 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.219378 6400 generic.go:356] "Generic (PLEG): container finished" podID="3e6b2116-c767-4c94-867f-448307c7bb4f" containerID="81fcf90a30ff68df4c9c736c87db8aab9504c54c5e147b737a9624d79469df03" exitCode=0 Nov 03 00:54:13.219484 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.219437 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" Nov 03 00:54:13.219678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.219485 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" event={"ID":"3e6b2116-c767-4c94-867f-448307c7bb4f","Type":"ContainerDied","Data":"81fcf90a30ff68df4c9c736c87db8aab9504c54c5e147b737a9624d79469df03"} Nov 03 00:54:13.219678 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.219531 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh" event={"ID":"3e6b2116-c767-4c94-867f-448307c7bb4f","Type":"ContainerDied","Data":"6276bbb512c7860176af6f14fa0bd18959a5ebcf3a33f90023ab96256ff14e62"} Nov 03 00:54:13.229888 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.229731 6400 scope.go:122] "RemoveContainer" containerID="a271f7385cc68aeb2e100db8c48e8c9b4e148bd999a56c2c1f3d102a647a0a26" Nov 03 00:54:13.230521 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:54:13.230500 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a271f7385cc68aeb2e100db8c48e8c9b4e148bd999a56c2c1f3d102a647a0a26\": container with ID starting with a271f7385cc68aeb2e100db8c48e8c9b4e148bd999a56c2c1f3d102a647a0a26 not found: ID does not exist" containerID="a271f7385cc68aeb2e100db8c48e8c9b4e148bd999a56c2c1f3d102a647a0a26" Nov 03 00:54:13.230635 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.230528 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a271f7385cc68aeb2e100db8c48e8c9b4e148bd999a56c2c1f3d102a647a0a26"} err="failed to get container status \"a271f7385cc68aeb2e100db8c48e8c9b4e148bd999a56c2c1f3d102a647a0a26\": rpc error: code = NotFound desc = could not find container \"a271f7385cc68aeb2e100db8c48e8c9b4e148bd999a56c2c1f3d102a647a0a26\": container with ID starting with a271f7385cc68aeb2e100db8c48e8c9b4e148bd999a56c2c1f3d102a647a0a26 not found: ID does not exist" Nov 03 00:54:13.230635 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.230547 6400 scope.go:122] "RemoveContainer" containerID="81fcf90a30ff68df4c9c736c87db8aab9504c54c5e147b737a9624d79469df03" Nov 03 00:54:13.240241 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.240217 6400 scope.go:122] "RemoveContainer" containerID="81fcf90a30ff68df4c9c736c87db8aab9504c54c5e147b737a9624d79469df03" Nov 03 00:54:13.240489 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:54:13.240473 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81fcf90a30ff68df4c9c736c87db8aab9504c54c5e147b737a9624d79469df03\": container with ID starting with 81fcf90a30ff68df4c9c736c87db8aab9504c54c5e147b737a9624d79469df03 not found: ID does not exist" containerID="81fcf90a30ff68df4c9c736c87db8aab9504c54c5e147b737a9624d79469df03" Nov 03 00:54:13.240569 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.240498 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81fcf90a30ff68df4c9c736c87db8aab9504c54c5e147b737a9624d79469df03"} err="failed to get container status \"81fcf90a30ff68df4c9c736c87db8aab9504c54c5e147b737a9624d79469df03\": rpc error: code = NotFound desc = could not find container \"81fcf90a30ff68df4c9c736c87db8aab9504c54c5e147b737a9624d79469df03\": container with ID starting with 81fcf90a30ff68df4c9c736c87db8aab9504c54c5e147b737a9624d79469df03 not found: ID does not exist" Nov 03 00:54:13.371994 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.371964 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh"] Nov 03 00:54:13.378375 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.378337 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-76tvh"] Nov 03 00:54:13.453786 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.453679 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:13.486952 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.486914 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92"] Nov 03 00:54:13.494477 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.494450 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-6jj92"] Nov 03 00:54:13.623216 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:13.623182 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:14.926472 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:14.926434 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:15.179125 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:15.179044 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d69e32-89cc-46ea-b72c-2e73e5bd17c4" path="/var/lib/kubelet/pods/11d69e32-89cc-46ea-b72c-2e73e5bd17c4/volumes" Nov 03 00:54:15.179698 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:15.179676 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6b2116-c767-4c94-867f-448307c7bb4f" path="/var/lib/kubelet/pods/3e6b2116-c767-4c94-867f-448307c7bb4f/volumes" Nov 03 00:54:15.523156 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:15.522996 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:18.596291 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:18.596252 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:20.197412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:20.197372 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:24.567883 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:24.567852 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:24.769082 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:24.769048 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:24.924899 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:24.924803 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:27.151692 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:54:27.151654 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:54:27.899651 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:27.899594 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" podUID="82c39d12-1304-4ced-a9ad-9163d91d6cbe" containerName="main" containerID="cri-o://b1722565b8ec72cba1d56a17bc86da07b35f04e1d3c21730e1fc68ab4ac2afdb" gracePeriod=45 Nov 03 00:54:28.023416 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.023377 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" podUID="82c39d12-1304-4ced-a9ad-9163d91d6cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": dial tcp 10.134.0.111:8000: connect: connection refused" Nov 03 00:54:28.149426 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.149399 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:54:28.266511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.266437 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-model-cache\") pod \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " Nov 03 00:54:28.266511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.266485 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-home\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-home\") pod \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " Nov 03 00:54:28.267055 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.266524 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tmp-dir\") pod \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " Nov 03 00:54:28.267055 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.266555 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-dshm\") pod \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " Nov 03 00:54:28.267055 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.266600 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tls-certs\" (UniqueName: \"kubernetes.io/secret/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tls-certs\") pod \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " Nov 03 00:54:28.267055 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.266722 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82c39d12-1304-4ced-a9ad-9163d91d6cbe-kube-api-access-29ksr\" (UniqueName: \"kubernetes.io/projected/82c39d12-1304-4ced-a9ad-9163d91d6cbe-kube-api-access-29ksr\") pod \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\" (UID: \"82c39d12-1304-4ced-a9ad-9163d91d6cbe\") " Nov 03 00:54:28.267055 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.266743 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-model-cache" pod "82c39d12-1304-4ced-a9ad-9163d91d6cbe" (UID: "82c39d12-1304-4ced-a9ad-9163d91d6cbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:28.267055 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.266843 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-home" pod "82c39d12-1304-4ced-a9ad-9163d91d6cbe" (UID: "82c39d12-1304-4ced-a9ad-9163d91d6cbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:28.267055 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.266851 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tmp-dir" pod "82c39d12-1304-4ced-a9ad-9163d91d6cbe" (UID: "82c39d12-1304-4ced-a9ad-9163d91d6cbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:28.268093 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.267442 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:28.268093 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.267466 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:28.268093 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.267480 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:28.269669 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.269646 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-dshm" pod "82c39d12-1304-4ced-a9ad-9163d91d6cbe" (UID: "82c39d12-1304-4ced-a9ad-9163d91d6cbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:28.269778 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.269714 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tls-certs" pod "82c39d12-1304-4ced-a9ad-9163d91d6cbe" (UID: "82c39d12-1304-4ced-a9ad-9163d91d6cbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:54:28.270164 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.270138 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c39d12-1304-4ced-a9ad-9163d91d6cbe-kube-api-access-29ksr" pod "82c39d12-1304-4ced-a9ad-9163d91d6cbe" (UID: "82c39d12-1304-4ced-a9ad-9163d91d6cbe"). InnerVolumeSpecName "kube-api-access-29ksr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:28.304537 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.304509 6400 generic.go:356] "Generic (PLEG): container finished" podID="82c39d12-1304-4ced-a9ad-9163d91d6cbe" containerID="b1722565b8ec72cba1d56a17bc86da07b35f04e1d3c21730e1fc68ab4ac2afdb" exitCode=0 Nov 03 00:54:28.304690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.304559 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" Nov 03 00:54:28.304690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.304594 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" event={"ID":"82c39d12-1304-4ced-a9ad-9163d91d6cbe","Type":"ContainerDied","Data":"b1722565b8ec72cba1d56a17bc86da07b35f04e1d3c21730e1fc68ab4ac2afdb"} Nov 03 00:54:28.304690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.304642 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw" event={"ID":"82c39d12-1304-4ced-a9ad-9163d91d6cbe","Type":"ContainerDied","Data":"b0c04342bbe14c1fa3b40b65f3e4f2febb251ccefe1ba34026c62273fb92d2dc"} Nov 03 00:54:28.304690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.304658 6400 scope.go:122] "RemoveContainer" containerID="b1722565b8ec72cba1d56a17bc86da07b35f04e1d3c21730e1fc68ab4ac2afdb" Nov 03 00:54:28.317918 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.317901 6400 scope.go:122] "RemoveContainer" containerID="b1722565b8ec72cba1d56a17bc86da07b35f04e1d3c21730e1fc68ab4ac2afdb" Nov 03 00:54:28.318216 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:54:28.318196 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1722565b8ec72cba1d56a17bc86da07b35f04e1d3c21730e1fc68ab4ac2afdb\": container with ID starting with b1722565b8ec72cba1d56a17bc86da07b35f04e1d3c21730e1fc68ab4ac2afdb not found: ID does not exist" containerID="b1722565b8ec72cba1d56a17bc86da07b35f04e1d3c21730e1fc68ab4ac2afdb" Nov 03 00:54:28.318276 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.318229 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1722565b8ec72cba1d56a17bc86da07b35f04e1d3c21730e1fc68ab4ac2afdb"} err="failed to get container status \"b1722565b8ec72cba1d56a17bc86da07b35f04e1d3c21730e1fc68ab4ac2afdb\": rpc error: code = NotFound desc = could not find container \"b1722565b8ec72cba1d56a17bc86da07b35f04e1d3c21730e1fc68ab4ac2afdb\": container with ID starting with b1722565b8ec72cba1d56a17bc86da07b35f04e1d3c21730e1fc68ab4ac2afdb not found: ID does not exist" Nov 03 00:54:28.368612 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.368590 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c39d12-1304-4ced-a9ad-9163d91d6cbe-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:28.368612 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.368611 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c39d12-1304-4ced-a9ad-9163d91d6cbe-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:28.368779 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.368620 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29ksr\" (UniqueName: \"kubernetes.io/projected/82c39d12-1304-4ced-a9ad-9163d91d6cbe-kube-api-access-29ksr\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:28.440128 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.440010 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:54:28.470369 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.470342 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw"] Nov 03 00:54:28.474051 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:28.474024 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-1f20dd/noten-model-1f20dd-kserve-84d8944c65-tkrbw"] Nov 03 00:54:29.179111 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:29.179083 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82c39d12-1304-4ced-a9ad-9163d91d6cbe" path="/var/lib/kubelet/pods/82c39d12-1304-4ced-a9ad-9163d91d6cbe/volumes" Nov 03 00:54:29.499234 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:29.499149 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:34.310387 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:34.310353 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:39.023987 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:39.023945 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:43.806420 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:43.806386 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:48.508673 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:48.508634 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:53.206508 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:53.206472 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:55.554074 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.554036 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:55.771561 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.771522 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7294-afa419"] Nov 03 00:54:55.772234 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.772214 6400 cpu_manager.go:414] "containerMap: removing container" podUID="11d69e32-89cc-46ea-b72c-2e73e5bd17c4" containerName="istio-proxy" Nov 03 00:54:55.772306 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.772238 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d69e32-89cc-46ea-b72c-2e73e5bd17c4" containerName="istio-proxy" Nov 03 00:54:55.772306 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.772282 6400 cpu_manager.go:414] "containerMap: removing container" podUID="3e6b2116-c767-4c94-867f-448307c7bb4f" containerName="istio-proxy" Nov 03 00:54:55.772306 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.772290 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6b2116-c767-4c94-867f-448307c7bb4f" containerName="istio-proxy" Nov 03 00:54:55.772402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.772315 6400 cpu_manager.go:414] "containerMap: removing container" podUID="82c39d12-1304-4ced-a9ad-9163d91d6cbe" containerName="main" Nov 03 00:54:55.772402 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.772322 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82c39d12-1304-4ced-a9ad-9163d91d6cbe" containerName="main" Nov 03 00:54:55.772473 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.772461 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="82c39d12-1304-4ced-a9ad-9163d91d6cbe" containerName="main" Nov 03 00:54:55.772516 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.772483 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6b2116-c767-4c94-867f-448307c7bb4f" containerName="istio-proxy" Nov 03 00:54:55.772516 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.772496 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d69e32-89cc-46ea-b72c-2e73e5bd17c4" containerName="istio-proxy" Nov 03 00:54:55.777561 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.777535 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-afa419" Nov 03 00:54:55.830677 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.830589 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7294-afa419"] Nov 03 00:54:55.865353 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.865319 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t49l\" (UniqueName: \"kubernetes.io/projected/b6d9b240-b90e-4b2a-813a-90a1a7fb312c-kube-api-access-8t49l\") pod \"test-curl-7294-afa419\" (UID: \"b6d9b240-b90e-4b2a-813a-90a1a7fb312c\") " pod="opendatahub/test-curl-7294-afa419" Nov 03 00:54:55.877293 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.877250 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fz96m\"" Nov 03 00:54:55.966578 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:55.966539 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t49l\" (UniqueName: \"kubernetes.io/projected/b6d9b240-b90e-4b2a-813a-90a1a7fb312c-kube-api-access-8t49l\") pod \"test-curl-7294-afa419\" (UID: \"b6d9b240-b90e-4b2a-813a-90a1a7fb312c\") " pod="opendatahub/test-curl-7294-afa419" Nov 03 00:54:56.022923 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:56.022895 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t49l\" (UniqueName: \"kubernetes.io/projected/b6d9b240-b90e-4b2a-813a-90a1a7fb312c-kube-api-access-8t49l\") pod \"test-curl-7294-afa419\" (UID: \"b6d9b240-b90e-4b2a-813a-90a1a7fb312c\") " pod="opendatahub/test-curl-7294-afa419" Nov 03 00:54:56.089352 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:56.089266 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-afa419" Nov 03 00:54:56.473471 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:56.473447 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7294-afa419"] Nov 03 00:54:56.476186 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:54:56.476162 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6d9b240_b90e_4b2a_813a_90a1a7fb312c.slice/crio-1b01a95f17e81aa5399c1e7deadcbff0b31ffc15665a84447a4487d27f002452 WatchSource:0}: Error finding container 1b01a95f17e81aa5399c1e7deadcbff0b31ffc15665a84447a4487d27f002452: Status 404 returned error can't find the container with id 1b01a95f17e81aa5399c1e7deadcbff0b31ffc15665a84447a4487d27f002452 Nov 03 00:54:57.153140 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:54:57.153093 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:54:57.469896 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:57.469786 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7294-afa419" event={"ID":"b6d9b240-b90e-4b2a-813a-90a1a7fb312c","Type":"ContainerStarted","Data":"1e2ac2ac4fc2d9ee881cbfc72ad7af42a766f135f864bc51861ae481392152c4"} Nov 03 00:54:57.469896 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:57.469853 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7294-afa419" event={"ID":"b6d9b240-b90e-4b2a-813a-90a1a7fb312c","Type":"ContainerStarted","Data":"1b01a95f17e81aa5399c1e7deadcbff0b31ffc15665a84447a4487d27f002452"} Nov 03 00:54:57.582016 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:57.581955 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7294-afa419" podStartSLOduration=2.58194048 podStartE2EDuration="2.58194048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:57.581169238 +0000 UTC m=+2944.532109397" watchObservedRunningTime="2026-11-03 00:54:57.58194048 +0000 UTC m=+2944.532880637" Nov 03 00:54:57.991628 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:57.991593 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:59.031600 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.031557 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7294-afa419"] Nov 03 00:54:59.034133 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.034095 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7294-afa419"] Nov 03 00:54:59.191968 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.191938 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7294-44f287"] Nov 03 00:54:59.192456 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.192441 6400 cpu_manager.go:414] "containerMap: removing container" podUID="b6d9b240-b90e-4b2a-813a-90a1a7fb312c" containerName="test-curl-7294-afa419" Nov 03 00:54:59.192508 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.192459 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d9b240-b90e-4b2a-813a-90a1a7fb312c" containerName="test-curl-7294-afa419" Nov 03 00:54:59.192547 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.192539 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d9b240-b90e-4b2a-813a-90a1a7fb312c" containerName="test-curl-7294-afa419" Nov 03 00:54:59.195904 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.195883 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-44f287" Nov 03 00:54:59.250673 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.250635 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7294-44f287"] Nov 03 00:54:59.301940 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.301906 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s82g\" (UniqueName: \"kubernetes.io/projected/732f76ed-283a-478e-b1b6-0cf5a822d32d-kube-api-access-5s82g\") pod \"test-curl-7294-44f287\" (UID: \"732f76ed-283a-478e-b1b6-0cf5a822d32d\") " pod="opendatahub/test-curl-7294-44f287" Nov 03 00:54:59.402834 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.402783 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s82g\" (UniqueName: \"kubernetes.io/projected/732f76ed-283a-478e-b1b6-0cf5a822d32d-kube-api-access-5s82g\") pod \"test-curl-7294-44f287\" (UID: \"732f76ed-283a-478e-b1b6-0cf5a822d32d\") " pod="opendatahub/test-curl-7294-44f287" Nov 03 00:54:59.459925 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.459886 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s82g\" (UniqueName: \"kubernetes.io/projected/732f76ed-283a-478e-b1b6-0cf5a822d32d-kube-api-access-5s82g\") pod \"test-curl-7294-44f287\" (UID: \"732f76ed-283a-478e-b1b6-0cf5a822d32d\") " pod="opendatahub/test-curl-7294-44f287" Nov 03 00:54:59.481444 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.481400 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7294-afa419" podUID="b6d9b240-b90e-4b2a-813a-90a1a7fb312c" containerName="test-curl-7294-afa419" containerID="cri-o://1e2ac2ac4fc2d9ee881cbfc72ad7af42a766f135f864bc51861ae481392152c4" gracePeriod=2 Nov 03 00:54:59.507640 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.507611 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-44f287" Nov 03 00:54:59.881115 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:54:59.881085 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7294-44f287"] Nov 03 00:54:59.883158 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:54:59.883126 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod732f76ed_283a_478e_b1b6_0cf5a822d32d.slice/crio-11318276c17ba20269fd638889bd141fcdde4b0d88a329b93befb0ba0cdd5573 WatchSource:0}: Error finding container 11318276c17ba20269fd638889bd141fcdde4b0d88a329b93befb0ba0cdd5573: Status 404 returned error can't find the container with id 11318276c17ba20269fd638889bd141fcdde4b0d88a329b93befb0ba0cdd5573 Nov 03 00:55:00.488658 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:00.488609 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7294-44f287" event={"ID":"732f76ed-283a-478e-b1b6-0cf5a822d32d","Type":"ContainerStarted","Data":"7d0b07b6e1beb00b987fa9fad4c6b6c1386e541b0539089e68cef055e95decf4"} Nov 03 00:55:00.488658 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:00.488663 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7294-44f287" event={"ID":"732f76ed-283a-478e-b1b6-0cf5a822d32d","Type":"ContainerStarted","Data":"11318276c17ba20269fd638889bd141fcdde4b0d88a329b93befb0ba0cdd5573"} Nov 03 00:55:00.600157 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:00.600096 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7294-44f287" podStartSLOduration=1.600076383 podStartE2EDuration="1.600076383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:00.599583211 +0000 UTC m=+2947.550523370" watchObservedRunningTime="2026-11-03 00:55:00.600076383 +0000 UTC m=+2947.551016541" Nov 03 00:55:01.697594 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:01.697569 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-afa419" Nov 03 00:55:01.833675 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:01.833577 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d9b240-b90e-4b2a-813a-90a1a7fb312c-kube-api-access-8t49l\" (UniqueName: \"kubernetes.io/projected/b6d9b240-b90e-4b2a-813a-90a1a7fb312c-kube-api-access-8t49l\") pod \"b6d9b240-b90e-4b2a-813a-90a1a7fb312c\" (UID: \"b6d9b240-b90e-4b2a-813a-90a1a7fb312c\") " Nov 03 00:55:01.835801 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:01.835770 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d9b240-b90e-4b2a-813a-90a1a7fb312c-kube-api-access-8t49l" pod "b6d9b240-b90e-4b2a-813a-90a1a7fb312c" (UID: "b6d9b240-b90e-4b2a-813a-90a1a7fb312c"). InnerVolumeSpecName "kube-api-access-8t49l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:01.934702 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:01.934657 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t49l\" (UniqueName: \"kubernetes.io/projected/b6d9b240-b90e-4b2a-813a-90a1a7fb312c-kube-api-access-8t49l\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:02.100369 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:02.100285 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7294-44f287"] Nov 03 00:55:02.104354 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:02.104317 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7294-44f287"] Nov 03 00:55:02.501853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:02.501745 6400 generic.go:356] "Generic (PLEG): container finished" podID="b6d9b240-b90e-4b2a-813a-90a1a7fb312c" containerID="1e2ac2ac4fc2d9ee881cbfc72ad7af42a766f135f864bc51861ae481392152c4" exitCode=137 Nov 03 00:55:02.501853 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:02.501799 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-afa419" Nov 03 00:55:02.502093 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:02.501860 6400 scope.go:122] "RemoveContainer" containerID="1e2ac2ac4fc2d9ee881cbfc72ad7af42a766f135f864bc51861ae481392152c4" Nov 03 00:55:02.502194 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:02.502140 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7294-44f287" podUID="732f76ed-283a-478e-b1b6-0cf5a822d32d" containerName="test-curl-7294-44f287" containerID="cri-o://7d0b07b6e1beb00b987fa9fad4c6b6c1386e541b0539089e68cef055e95decf4" gracePeriod=2 Nov 03 00:55:02.512634 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:02.512609 6400 scope.go:122] "RemoveContainer" containerID="1e2ac2ac4fc2d9ee881cbfc72ad7af42a766f135f864bc51861ae481392152c4" Nov 03 00:55:02.513025 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:02.513003 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2ac2ac4fc2d9ee881cbfc72ad7af42a766f135f864bc51861ae481392152c4\": container with ID starting with 1e2ac2ac4fc2d9ee881cbfc72ad7af42a766f135f864bc51861ae481392152c4 not found: ID does not exist" containerID="1e2ac2ac4fc2d9ee881cbfc72ad7af42a766f135f864bc51861ae481392152c4" Nov 03 00:55:02.513138 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:02.513035 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2ac2ac4fc2d9ee881cbfc72ad7af42a766f135f864bc51861ae481392152c4"} err="failed to get container status \"1e2ac2ac4fc2d9ee881cbfc72ad7af42a766f135f864bc51861ae481392152c4\": rpc error: code = NotFound desc = could not find container \"1e2ac2ac4fc2d9ee881cbfc72ad7af42a766f135f864bc51861ae481392152c4\": container with ID starting with 1e2ac2ac4fc2d9ee881cbfc72ad7af42a766f135f864bc51861ae481392152c4 not found: ID does not exist" Nov 03 00:55:02.553004 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:02.552958 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7294-44f287\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="732f76ed-283a-478e-b1b6-0cf5a822d32d" pod="opendatahub/test-curl-7294-44f287" Nov 03 00:55:02.603691 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:02.603648 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7294-44f287\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="732f76ed-283a-478e-b1b6-0cf5a822d32d" pod="opendatahub/test-curl-7294-44f287" Nov 03 00:55:02.654524 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:02.654477 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7294-afa419\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="b6d9b240-b90e-4b2a-813a-90a1a7fb312c" pod="opendatahub/test-curl-7294-afa419" Nov 03 00:55:02.712030 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:02.711994 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:02.976493 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:02.976451 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:03.178952 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.178916 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d9b240-b90e-4b2a-813a-90a1a7fb312c" path="/var/lib/kubelet/pods/b6d9b240-b90e-4b2a-813a-90a1a7fb312c/volumes" Nov 03 00:55:03.226105 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:03.226067 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7294-44f287\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="732f76ed-283a-478e-b1b6-0cf5a822d32d" pod="opendatahub/test-curl-7294-44f287" Nov 03 00:55:03.276979 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:03.276878 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7294-afa419\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="b6d9b240-b90e-4b2a-813a-90a1a7fb312c" pod="opendatahub/test-curl-7294-afa419" Nov 03 00:55:03.468279 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.468241 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw"] Nov 03 00:55:03.468799 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.468781 6400 cpu_manager.go:414] "containerMap: removing container" podUID="732f76ed-283a-478e-b1b6-0cf5a822d32d" containerName="test-curl-7294-44f287" Nov 03 00:55:03.468945 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.468801 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732f76ed-283a-478e-b1b6-0cf5a822d32d" containerName="test-curl-7294-44f287" Nov 03 00:55:03.469000 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.468970 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="732f76ed-283a-478e-b1b6-0cf5a822d32d" containerName="test-curl-7294-44f287" Nov 03 00:55:03.479346 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.479307 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.552848 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.550360 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-f40c6c-openshift-default-dockercfg-6tw9h\"" Nov 03 00:55:03.557857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.557355 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-socket\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.557857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.557411 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-data\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.557857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.557456 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27d8659-f856-403f-a9eb-d86f17f24506-istiod-ca-cert\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.557857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.557505 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-credential-socket\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.557857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.557622 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27d8659-f856-403f-a9eb-d86f17f24506-istio-podinfo\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.557857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.557651 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-envoy\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.557857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.557674 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-certs\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.557857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.557694 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-istio-token\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.557857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.557719 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqx8d\" (UniqueName: \"kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-kube-api-access-qqx8d\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.575934 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.575904 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:03.602169 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.602129 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw"] Nov 03 00:55:03.659298 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.659262 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-istio-token\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.659503 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.659316 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqx8d\" (UniqueName: \"kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-kube-api-access-qqx8d\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.659503 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.659394 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-socket\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.659503 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.659413 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-data\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.659655 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.659549 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27d8659-f856-403f-a9eb-d86f17f24506-istiod-ca-cert\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.659655 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.659596 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-credential-socket\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.659782 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.659697 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27d8659-f856-403f-a9eb-d86f17f24506-istio-podinfo\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.659782 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.659741 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-envoy\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.659782 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.659774 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-certs\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.660001 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.659846 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-data\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.660001 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.659975 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-socket\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.660110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.660027 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-credential-socket\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.660110 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.660100 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-certs\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.660338 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.660317 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27d8659-f856-403f-a9eb-d86f17f24506-istiod-ca-cert\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.662208 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.662184 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-envoy\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.662535 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.662519 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27d8659-f856-403f-a9eb-d86f17f24506-istio-podinfo\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.716086 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.716055 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-istio-token\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.716461 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.716128 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqx8d\" (UniqueName: \"kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-kube-api-access-qqx8d\") pod \"e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:03.806748 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:03.806665 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:04.131221 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:04.131120 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:04.240962 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:04.240937 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw"] Nov 03 00:55:04.243317 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:55:04.243283 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27d8659_f856_403f_a9eb_d86f17f24506.slice/crio-11b26b42e5db47442e92f737dc14da5de1c2f204d1e9aba93d340cea4cca2843 WatchSource:0}: Error finding container 11b26b42e5db47442e92f737dc14da5de1c2f204d1e9aba93d340cea4cca2843: Status 404 returned error can't find the container with id 11b26b42e5db47442e92f737dc14da5de1c2f204d1e9aba93d340cea4cca2843 Nov 03 00:55:04.245907 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:04.245877 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:55:04.245978 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:04.245946 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:55:04.246016 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:04.245976 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:55:04.525911 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:04.525861 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" event={"ID":"e27d8659-f856-403f-a9eb-d86f17f24506","Type":"ContainerStarted","Data":"dc8ed8f15fe481d9f4ce9a6f4a210a69b6629d3b074c5158144f0a49a5df2434"} Nov 03 00:55:04.526076 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:04.525950 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" event={"ID":"e27d8659-f856-403f-a9eb-d86f17f24506","Type":"ContainerStarted","Data":"11b26b42e5db47442e92f737dc14da5de1c2f204d1e9aba93d340cea4cca2843"} Nov 03 00:55:04.641917 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:04.641836 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" podStartSLOduration=1.6417917229999999 podStartE2EDuration="1.641791723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:04.641349638 +0000 UTC m=+2951.592289791" watchObservedRunningTime="2026-11-03 00:55:04.641791723 +0000 UTC m=+2951.592731881" Nov 03 00:55:04.745226 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:04.745200 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-44f287" Nov 03 00:55:04.807094 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:04.807056 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:04.874861 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:04.874801 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732f76ed-283a-478e-b1b6-0cf5a822d32d-kube-api-access-5s82g\" (UniqueName: \"kubernetes.io/projected/732f76ed-283a-478e-b1b6-0cf5a822d32d-kube-api-access-5s82g\") pod \"732f76ed-283a-478e-b1b6-0cf5a822d32d\" (UID: \"732f76ed-283a-478e-b1b6-0cf5a822d32d\") " Nov 03 00:55:04.877270 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:04.877233 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732f76ed-283a-478e-b1b6-0cf5a822d32d-kube-api-access-5s82g" pod "732f76ed-283a-478e-b1b6-0cf5a822d32d" (UID: "732f76ed-283a-478e-b1b6-0cf5a822d32d"). InnerVolumeSpecName "kube-api-access-5s82g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:04.976433 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:04.976389 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s82g\" (UniqueName: \"kubernetes.io/projected/732f76ed-283a-478e-b1b6-0cf5a822d32d-kube-api-access-5s82g\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:05.179196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:05.179077 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732f76ed-283a-478e-b1b6-0cf5a822d32d" path="/var/lib/kubelet/pods/732f76ed-283a-478e-b1b6-0cf5a822d32d/volumes" Nov 03 00:55:05.532167 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:05.532082 6400 generic.go:356] "Generic (PLEG): container finished" podID="732f76ed-283a-478e-b1b6-0cf5a822d32d" containerID="7d0b07b6e1beb00b987fa9fad4c6b6c1386e541b0539089e68cef055e95decf4" exitCode=137 Nov 03 00:55:05.532167 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:05.532138 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-44f287" Nov 03 00:55:05.532371 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:05.532178 6400 scope.go:122] "RemoveContainer" containerID="7d0b07b6e1beb00b987fa9fad4c6b6c1386e541b0539089e68cef055e95decf4" Nov 03 00:55:05.541256 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:05.541233 6400 scope.go:122] "RemoveContainer" containerID="7d0b07b6e1beb00b987fa9fad4c6b6c1386e541b0539089e68cef055e95decf4" Nov 03 00:55:05.541593 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:05.541573 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0b07b6e1beb00b987fa9fad4c6b6c1386e541b0539089e68cef055e95decf4\": container with ID starting with 7d0b07b6e1beb00b987fa9fad4c6b6c1386e541b0539089e68cef055e95decf4 not found: ID does not exist" containerID="7d0b07b6e1beb00b987fa9fad4c6b6c1386e541b0539089e68cef055e95decf4" Nov 03 00:55:05.541640 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:05.541604 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0b07b6e1beb00b987fa9fad4c6b6c1386e541b0539089e68cef055e95decf4"} err="failed to get container status \"7d0b07b6e1beb00b987fa9fad4c6b6c1386e541b0539089e68cef055e95decf4\": rpc error: code = NotFound desc = could not find container \"7d0b07b6e1beb00b987fa9fad4c6b6c1386e541b0539089e68cef055e95decf4\": container with ID starting with 7d0b07b6e1beb00b987fa9fad4c6b6c1386e541b0539089e68cef055e95decf4 not found: ID does not exist" Nov 03 00:55:05.814339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:05.814306 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:06.539137 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:06.539105 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:06.540237 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:06.540215 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:55:07.570105 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:07.570059 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:07.741757 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:07.741726 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:07.921391 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:07.921279 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:10.788906 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:10.788872 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:10.962609 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:10.962576 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:12.609117 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:12.609084 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:15.266199 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:15.266162 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:17.313292 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:17.313257 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:20.523659 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:20.523621 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:55:22.009513 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:22.009477 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:26.719538 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:26.719499 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:27.152959 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:27.152924 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:55:31.439256 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:31.439162 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:31.881019 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:31.880981 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp"] Nov 03 00:55:31.915715 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:31.915679 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:33.406242 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:33.406191 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:33.607352 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:33.607314 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7297-d4d5d5"] Nov 03 00:55:33.611271 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:33.611251 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-d4d5d5" Nov 03 00:55:33.664474 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:33.664389 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7297-d4d5d5"] Nov 03 00:55:33.711248 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:33.711210 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fz96m\"" Nov 03 00:55:33.735078 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:33.735039 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ln6h\" (UniqueName: \"kubernetes.io/projected/cff417a0-9fef-4700-b0b5-3631eae637eb-kube-api-access-2ln6h\") pod \"test-curl-7297-d4d5d5\" (UID: \"cff417a0-9fef-4700-b0b5-3631eae637eb\") " pod="opendatahub/test-curl-7297-d4d5d5" Nov 03 00:55:33.836266 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:33.836227 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ln6h\" (UniqueName: \"kubernetes.io/projected/cff417a0-9fef-4700-b0b5-3631eae637eb-kube-api-access-2ln6h\") pod \"test-curl-7297-d4d5d5\" (UID: \"cff417a0-9fef-4700-b0b5-3631eae637eb\") " pod="opendatahub/test-curl-7297-d4d5d5" Nov 03 00:55:33.893524 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:33.893492 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ln6h\" (UniqueName: \"kubernetes.io/projected/cff417a0-9fef-4700-b0b5-3631eae637eb-kube-api-access-2ln6h\") pod \"test-curl-7297-d4d5d5\" (UID: \"cff417a0-9fef-4700-b0b5-3631eae637eb\") " pod="opendatahub/test-curl-7297-d4d5d5" Nov 03 00:55:33.923517 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:33.923442 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-d4d5d5" Nov 03 00:55:34.302037 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:34.302011 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7297-d4d5d5"] Nov 03 00:55:34.305051 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:55:34.305022 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcff417a0_9fef_4700_b0b5_3631eae637eb.slice/crio-e12d03bdca69b93ebf8e02940c5df9404f619299f8c43ba4eacb1c592414a13e WatchSource:0}: Error finding container e12d03bdca69b93ebf8e02940c5df9404f619299f8c43ba4eacb1c592414a13e: Status 404 returned error can't find the container with id e12d03bdca69b93ebf8e02940c5df9404f619299f8c43ba4eacb1c592414a13e Nov 03 00:55:34.694092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:34.693986 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7297-d4d5d5" event={"ID":"cff417a0-9fef-4700-b0b5-3631eae637eb","Type":"ContainerStarted","Data":"48c3ea44bf030eb9a8604145360a9218e1134445c7fbec335ecc40d3f6175d92"} Nov 03 00:55:34.694092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:34.694033 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7297-d4d5d5" event={"ID":"cff417a0-9fef-4700-b0b5-3631eae637eb","Type":"ContainerStarted","Data":"e12d03bdca69b93ebf8e02940c5df9404f619299f8c43ba4eacb1c592414a13e"} Nov 03 00:55:34.805639 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:34.805581 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7297-d4d5d5" podStartSLOduration=1.805567134 podStartE2EDuration="1.805567134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:34.804547617 +0000 UTC m=+2981.755487777" watchObservedRunningTime="2026-11-03 00:55:34.805567134 +0000 UTC m=+2981.756507293" Nov 03 00:55:36.190447 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.190409 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7297-d4d5d5"] Nov 03 00:55:36.193911 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.193876 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7297-d4d5d5"] Nov 03 00:55:36.259755 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.259550 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:36.344594 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.344557 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7297-34c2c1"] Nov 03 00:55:36.345166 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.345147 6400 cpu_manager.go:414] "containerMap: removing container" podUID="cff417a0-9fef-4700-b0b5-3631eae637eb" containerName="test-curl-7297-d4d5d5" Nov 03 00:55:36.345276 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.345168 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff417a0-9fef-4700-b0b5-3631eae637eb" containerName="test-curl-7297-d4d5d5" Nov 03 00:55:36.345339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.345287 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff417a0-9fef-4700-b0b5-3631eae637eb" containerName="test-curl-7297-d4d5d5" Nov 03 00:55:36.348666 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.348640 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-34c2c1" Nov 03 00:55:36.400927 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.400893 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7297-34c2c1"] Nov 03 00:55:36.466477 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.466363 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrw8t\" (UniqueName: \"kubernetes.io/projected/549c43d3-e05b-45cd-99b8-70e32651e1e1-kube-api-access-nrw8t\") pod \"test-curl-7297-34c2c1\" (UID: \"549c43d3-e05b-45cd-99b8-70e32651e1e1\") " pod="opendatahub/test-curl-7297-34c2c1" Nov 03 00:55:36.568180 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.568130 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrw8t\" (UniqueName: \"kubernetes.io/projected/549c43d3-e05b-45cd-99b8-70e32651e1e1-kube-api-access-nrw8t\") pod \"test-curl-7297-34c2c1\" (UID: \"549c43d3-e05b-45cd-99b8-70e32651e1e1\") " pod="opendatahub/test-curl-7297-34c2c1" Nov 03 00:55:36.625045 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.625001 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrw8t\" (UniqueName: \"kubernetes.io/projected/549c43d3-e05b-45cd-99b8-70e32651e1e1-kube-api-access-nrw8t\") pod \"test-curl-7297-34c2c1\" (UID: \"549c43d3-e05b-45cd-99b8-70e32651e1e1\") " pod="opendatahub/test-curl-7297-34c2c1" Nov 03 00:55:36.661954 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.661911 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-34c2c1" Nov 03 00:55:36.705918 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:36.705861 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7297-d4d5d5" podUID="cff417a0-9fef-4700-b0b5-3631eae637eb" containerName="test-curl-7297-d4d5d5" containerID="cri-o://48c3ea44bf030eb9a8604145360a9218e1134445c7fbec335ecc40d3f6175d92" gracePeriod=2 Nov 03 00:55:36.756837 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:36.756775 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-d4d5d5\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="cff417a0-9fef-4700-b0b5-3631eae637eb" pod="opendatahub/test-curl-7297-d4d5d5" Nov 03 00:55:37.041695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:37.041664 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7297-34c2c1"] Nov 03 00:55:37.043381 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:55:37.043346 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod549c43d3_e05b_45cd_99b8_70e32651e1e1.slice/crio-b90c734469f5e497813441abc88d73980ab3e6fffcb39db5952536b82d09217c WatchSource:0}: Error finding container b90c734469f5e497813441abc88d73980ab3e6fffcb39db5952536b82d09217c: Status 404 returned error can't find the container with id b90c734469f5e497813441abc88d73980ab3e6fffcb39db5952536b82d09217c Nov 03 00:55:37.713230 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:37.713197 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7297-34c2c1" event={"ID":"549c43d3-e05b-45cd-99b8-70e32651e1e1","Type":"ContainerStarted","Data":"4a61055765bec443f86569b23ad33f18dd00a8beff0dcec682479fb0e05d5a55"} Nov 03 00:55:37.713230 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:37.713238 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7297-34c2c1" event={"ID":"549c43d3-e05b-45cd-99b8-70e32651e1e1","Type":"ContainerStarted","Data":"b90c734469f5e497813441abc88d73980ab3e6fffcb39db5952536b82d09217c"} Nov 03 00:55:37.823688 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:37.823625 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7297-34c2c1" podStartSLOduration=1.8236051020000001 podStartE2EDuration="1.823605102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:37.823400027 +0000 UTC m=+2984.774340186" watchObservedRunningTime="2026-11-03 00:55:37.823605102 +0000 UTC m=+2984.774545261" Nov 03 00:55:38.918558 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:38.918526 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-d4d5d5" Nov 03 00:55:38.991991 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:38.991907 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff417a0-9fef-4700-b0b5-3631eae637eb-kube-api-access-2ln6h\" (UniqueName: \"kubernetes.io/projected/cff417a0-9fef-4700-b0b5-3631eae637eb-kube-api-access-2ln6h\") pod \"cff417a0-9fef-4700-b0b5-3631eae637eb\" (UID: \"cff417a0-9fef-4700-b0b5-3631eae637eb\") " Nov 03 00:55:38.994175 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:38.994144 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff417a0-9fef-4700-b0b5-3631eae637eb-kube-api-access-2ln6h" pod "cff417a0-9fef-4700-b0b5-3631eae637eb" (UID: "cff417a0-9fef-4700-b0b5-3631eae637eb"). InnerVolumeSpecName "kube-api-access-2ln6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:39.092762 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.092727 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ln6h\" (UniqueName: \"kubernetes.io/projected/cff417a0-9fef-4700-b0b5-3631eae637eb-kube-api-access-2ln6h\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:39.180010 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.179973 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff417a0-9fef-4700-b0b5-3631eae637eb" path="/var/lib/kubelet/pods/cff417a0-9fef-4700-b0b5-3631eae637eb/volumes" Nov 03 00:55:39.286314 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.286276 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7297-34c2c1"] Nov 03 00:55:39.290111 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.290075 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7297-34c2c1"] Nov 03 00:55:39.469673 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.469638 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7297-2b3c6a"] Nov 03 00:55:39.470148 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.470135 6400 cpu_manager.go:414] "containerMap: removing container" podUID="549c43d3-e05b-45cd-99b8-70e32651e1e1" containerName="test-curl-7297-34c2c1" Nov 03 00:55:39.470208 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.470150 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549c43d3-e05b-45cd-99b8-70e32651e1e1" containerName="test-curl-7297-34c2c1" Nov 03 00:55:39.470244 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.470239 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="549c43d3-e05b-45cd-99b8-70e32651e1e1" containerName="test-curl-7297-34c2c1" Nov 03 00:55:39.473597 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.473581 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-2b3c6a" Nov 03 00:55:39.497105 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.497073 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z85qj\" (UniqueName: \"kubernetes.io/projected/59e7db63-1f77-4c20-98c5-dae6612267a2-kube-api-access-z85qj\") pod \"test-curl-7297-2b3c6a\" (UID: \"59e7db63-1f77-4c20-98c5-dae6612267a2\") " pod="opendatahub/test-curl-7297-2b3c6a" Nov 03 00:55:39.532231 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.532196 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7297-2b3c6a"] Nov 03 00:55:39.598802 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.598716 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z85qj\" (UniqueName: \"kubernetes.io/projected/59e7db63-1f77-4c20-98c5-dae6612267a2-kube-api-access-z85qj\") pod \"test-curl-7297-2b3c6a\" (UID: \"59e7db63-1f77-4c20-98c5-dae6612267a2\") " pod="opendatahub/test-curl-7297-2b3c6a" Nov 03 00:55:39.655417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.655373 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z85qj\" (UniqueName: \"kubernetes.io/projected/59e7db63-1f77-4c20-98c5-dae6612267a2-kube-api-access-z85qj\") pod \"test-curl-7297-2b3c6a\" (UID: \"59e7db63-1f77-4c20-98c5-dae6612267a2\") " pod="opendatahub/test-curl-7297-2b3c6a" Nov 03 00:55:39.726912 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.726874 6400 generic.go:356] "Generic (PLEG): container finished" podID="cff417a0-9fef-4700-b0b5-3631eae637eb" containerID="48c3ea44bf030eb9a8604145360a9218e1134445c7fbec335ecc40d3f6175d92" exitCode=137 Nov 03 00:55:39.727092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.726921 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-d4d5d5" Nov 03 00:55:39.727092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.726978 6400 scope.go:122] "RemoveContainer" containerID="48c3ea44bf030eb9a8604145360a9218e1134445c7fbec335ecc40d3f6175d92" Nov 03 00:55:39.727092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.727059 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7297-34c2c1" podUID="549c43d3-e05b-45cd-99b8-70e32651e1e1" containerName="test-curl-7297-34c2c1" containerID="cri-o://4a61055765bec443f86569b23ad33f18dd00a8beff0dcec682479fb0e05d5a55" gracePeriod=2 Nov 03 00:55:39.736200 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.736175 6400 scope.go:122] "RemoveContainer" containerID="48c3ea44bf030eb9a8604145360a9218e1134445c7fbec335ecc40d3f6175d92" Nov 03 00:55:39.736547 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:39.736521 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c3ea44bf030eb9a8604145360a9218e1134445c7fbec335ecc40d3f6175d92\": container with ID starting with 48c3ea44bf030eb9a8604145360a9218e1134445c7fbec335ecc40d3f6175d92 not found: ID does not exist" containerID="48c3ea44bf030eb9a8604145360a9218e1134445c7fbec335ecc40d3f6175d92" Nov 03 00:55:39.736646 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.736559 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c3ea44bf030eb9a8604145360a9218e1134445c7fbec335ecc40d3f6175d92"} err="failed to get container status \"48c3ea44bf030eb9a8604145360a9218e1134445c7fbec335ecc40d3f6175d92\": rpc error: code = NotFound desc = could not find container \"48c3ea44bf030eb9a8604145360a9218e1134445c7fbec335ecc40d3f6175d92\": container with ID starting with 48c3ea44bf030eb9a8604145360a9218e1134445c7fbec335ecc40d3f6175d92 not found: ID does not exist" Nov 03 00:55:39.777964 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:39.777927 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-34c2c1\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="549c43d3-e05b-45cd-99b8-70e32651e1e1" pod="opendatahub/test-curl-7297-34c2c1" Nov 03 00:55:39.785412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:39.785386 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-2b3c6a" Nov 03 00:55:39.828629 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:39.828590 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-34c2c1\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="549c43d3-e05b-45cd-99b8-70e32651e1e1" pod="opendatahub/test-curl-7297-34c2c1" Nov 03 00:55:40.164448 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:40.164421 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7297-2b3c6a"] Nov 03 00:55:40.167147 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:55:40.167108 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59e7db63_1f77_4c20_98c5_dae6612267a2.slice/crio-9e54ddd22d79c46b61734324200c6bb0e2049a6aa6a4f788f453b6195971251a WatchSource:0}: Error finding container 9e54ddd22d79c46b61734324200c6bb0e2049a6aa6a4f788f453b6195971251a: Status 404 returned error can't find the container with id 9e54ddd22d79c46b61734324200c6bb0e2049a6aa6a4f788f453b6195971251a Nov 03 00:55:40.734926 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:40.734890 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7297-2b3c6a" event={"ID":"59e7db63-1f77-4c20-98c5-dae6612267a2","Type":"ContainerStarted","Data":"2ae8d3833e501353be69057ba219f3bb39c3e07e4c689a9b08dedac0ca3f1bb2"} Nov 03 00:55:40.734926 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:40.734938 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7297-2b3c6a" event={"ID":"59e7db63-1f77-4c20-98c5-dae6612267a2","Type":"ContainerStarted","Data":"9e54ddd22d79c46b61734324200c6bb0e2049a6aa6a4f788f453b6195971251a"} Nov 03 00:55:40.785745 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:40.785692 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-34c2c1\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="549c43d3-e05b-45cd-99b8-70e32651e1e1" pod="opendatahub/test-curl-7297-34c2c1" Nov 03 00:55:40.896397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:40.896329 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7297-2b3c6a" podStartSLOduration=1.8963111499999998 podStartE2EDuration="1.89631115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:40.895399062 +0000 UTC m=+2987.846339220" watchObservedRunningTime="2026-11-03 00:55:40.89631115 +0000 UTC m=+2987.847251309" Nov 03 00:55:40.999455 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:40.999328 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:41.968112 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:41.968085 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-34c2c1" Nov 03 00:55:42.023869 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.023752 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549c43d3-e05b-45cd-99b8-70e32651e1e1-kube-api-access-nrw8t\" (UniqueName: \"kubernetes.io/projected/549c43d3-e05b-45cd-99b8-70e32651e1e1-kube-api-access-nrw8t\") pod \"549c43d3-e05b-45cd-99b8-70e32651e1e1\" (UID: \"549c43d3-e05b-45cd-99b8-70e32651e1e1\") " Nov 03 00:55:42.026135 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.026106 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549c43d3-e05b-45cd-99b8-70e32651e1e1-kube-api-access-nrw8t" pod "549c43d3-e05b-45cd-99b8-70e32651e1e1" (UID: "549c43d3-e05b-45cd-99b8-70e32651e1e1"). InnerVolumeSpecName "kube-api-access-nrw8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:42.125690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.125641 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrw8t\" (UniqueName: \"kubernetes.io/projected/549c43d3-e05b-45cd-99b8-70e32651e1e1-kube-api-access-nrw8t\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:42.292196 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.292162 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7297-2b3c6a"] Nov 03 00:55:42.295725 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.295697 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7297-2b3c6a"] Nov 03 00:55:42.460287 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.460252 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7297-e64e9e"] Nov 03 00:55:42.460760 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.460746 6400 cpu_manager.go:414] "containerMap: removing container" podUID="59e7db63-1f77-4c20-98c5-dae6612267a2" containerName="test-curl-7297-2b3c6a" Nov 03 00:55:42.460805 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.460762 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e7db63-1f77-4c20-98c5-dae6612267a2" containerName="test-curl-7297-2b3c6a" Nov 03 00:55:42.460909 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.460899 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e7db63-1f77-4c20-98c5-dae6612267a2" containerName="test-curl-7297-2b3c6a" Nov 03 00:55:42.464076 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.464059 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-e64e9e" Nov 03 00:55:42.517881 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.517846 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7297-e64e9e"] Nov 03 00:55:42.530749 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.530711 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42rnr\" (UniqueName: \"kubernetes.io/projected/4cafd7d0-e316-47e7-a835-ef0e805e2d56-kube-api-access-42rnr\") pod \"test-curl-7297-e64e9e\" (UID: \"4cafd7d0-e316-47e7-a835-ef0e805e2d56\") " pod="opendatahub/test-curl-7297-e64e9e" Nov 03 00:55:42.631954 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.631861 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42rnr\" (UniqueName: \"kubernetes.io/projected/4cafd7d0-e316-47e7-a835-ef0e805e2d56-kube-api-access-42rnr\") pod \"test-curl-7297-e64e9e\" (UID: \"4cafd7d0-e316-47e7-a835-ef0e805e2d56\") " pod="opendatahub/test-curl-7297-e64e9e" Nov 03 00:55:42.688499 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.688471 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42rnr\" (UniqueName: \"kubernetes.io/projected/4cafd7d0-e316-47e7-a835-ef0e805e2d56-kube-api-access-42rnr\") pod \"test-curl-7297-e64e9e\" (UID: \"4cafd7d0-e316-47e7-a835-ef0e805e2d56\") " pod="opendatahub/test-curl-7297-e64e9e" Nov 03 00:55:42.746881 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.746844 6400 generic.go:356] "Generic (PLEG): container finished" podID="549c43d3-e05b-45cd-99b8-70e32651e1e1" containerID="4a61055765bec443f86569b23ad33f18dd00a8beff0dcec682479fb0e05d5a55" exitCode=137 Nov 03 00:55:42.747050 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.746894 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-34c2c1" Nov 03 00:55:42.747050 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.746978 6400 scope.go:122] "RemoveContainer" containerID="4a61055765bec443f86569b23ad33f18dd00a8beff0dcec682479fb0e05d5a55" Nov 03 00:55:42.747261 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.747242 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7297-2b3c6a" podUID="59e7db63-1f77-4c20-98c5-dae6612267a2" containerName="test-curl-7297-2b3c6a" containerID="cri-o://2ae8d3833e501353be69057ba219f3bb39c3e07e4c689a9b08dedac0ca3f1bb2" gracePeriod=2 Nov 03 00:55:42.756996 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.756976 6400 scope.go:122] "RemoveContainer" containerID="4a61055765bec443f86569b23ad33f18dd00a8beff0dcec682479fb0e05d5a55" Nov 03 00:55:42.757319 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:42.757298 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a61055765bec443f86569b23ad33f18dd00a8beff0dcec682479fb0e05d5a55\": container with ID starting with 4a61055765bec443f86569b23ad33f18dd00a8beff0dcec682479fb0e05d5a55 not found: ID does not exist" containerID="4a61055765bec443f86569b23ad33f18dd00a8beff0dcec682479fb0e05d5a55" Nov 03 00:55:42.757389 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.757330 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a61055765bec443f86569b23ad33f18dd00a8beff0dcec682479fb0e05d5a55"} err="failed to get container status \"4a61055765bec443f86569b23ad33f18dd00a8beff0dcec682479fb0e05d5a55\": rpc error: code = NotFound desc = could not find container \"4a61055765bec443f86569b23ad33f18dd00a8beff0dcec682479fb0e05d5a55\": container with ID starting with 4a61055765bec443f86569b23ad33f18dd00a8beff0dcec682479fb0e05d5a55 not found: ID does not exist" Nov 03 00:55:42.776851 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:42.776789 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-e64e9e" Nov 03 00:55:42.797896 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:42.797786 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-2b3c6a\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="59e7db63-1f77-4c20-98c5-dae6612267a2" pod="opendatahub/test-curl-7297-2b3c6a" Nov 03 00:55:42.848394 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:42.848345 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-34c2c1\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="549c43d3-e05b-45cd-99b8-70e32651e1e1" pod="opendatahub/test-curl-7297-34c2c1" Nov 03 00:55:42.899156 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:42.899059 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-2b3c6a\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="59e7db63-1f77-4c20-98c5-dae6612267a2" pod="opendatahub/test-curl-7297-2b3c6a" Nov 03 00:55:43.152610 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:43.152559 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7297-e64e9e"] Nov 03 00:55:43.155376 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:55:43.155349 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cafd7d0_e316_47e7_a835_ef0e805e2d56.slice/crio-57f8e747a5a746af5cb87f87b4d8d68f3cec7e5d25bd4b62aec28a55bf0eba35 WatchSource:0}: Error finding container 57f8e747a5a746af5cb87f87b4d8d68f3cec7e5d25bd4b62aec28a55bf0eba35: Status 404 returned error can't find the container with id 57f8e747a5a746af5cb87f87b4d8d68f3cec7e5d25bd4b62aec28a55bf0eba35 Nov 03 00:55:43.180503 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:43.180412 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549c43d3-e05b-45cd-99b8-70e32651e1e1" path="/var/lib/kubelet/pods/549c43d3-e05b-45cd-99b8-70e32651e1e1/volumes" Nov 03 00:55:43.227084 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:43.227054 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-34c2c1\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="549c43d3-e05b-45cd-99b8-70e32651e1e1" pod="opendatahub/test-curl-7297-34c2c1" Nov 03 00:55:43.277702 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:43.277652 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-2b3c6a\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="59e7db63-1f77-4c20-98c5-dae6612267a2" pod="opendatahub/test-curl-7297-2b3c6a" Nov 03 00:55:43.755630 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:43.755591 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7297-e64e9e" event={"ID":"4cafd7d0-e316-47e7-a835-ef0e805e2d56","Type":"ContainerStarted","Data":"9f9221cbb330aa8bd988b7721f6ca29d7a51403e4fa0843bfadeac34f3a20a9b"} Nov 03 00:55:43.755630 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:43.755635 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7297-e64e9e" event={"ID":"4cafd7d0-e316-47e7-a835-ef0e805e2d56","Type":"ContainerStarted","Data":"57f8e747a5a746af5cb87f87b4d8d68f3cec7e5d25bd4b62aec28a55bf0eba35"} Nov 03 00:55:43.867685 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:43.867617 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7297-e64e9e" podStartSLOduration=1.867597302 podStartE2EDuration="1.867597302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:43.865816151 +0000 UTC m=+2990.816756309" watchObservedRunningTime="2026-11-03 00:55:43.867597302 +0000 UTC m=+2990.818537463" Nov 03 00:55:44.960403 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:44.960363 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-2b3c6a" Nov 03 00:55:45.057340 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:45.057299 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e7db63-1f77-4c20-98c5-dae6612267a2-kube-api-access-z85qj\" (UniqueName: \"kubernetes.io/projected/59e7db63-1f77-4c20-98c5-dae6612267a2-kube-api-access-z85qj\") pod \"59e7db63-1f77-4c20-98c5-dae6612267a2\" (UID: \"59e7db63-1f77-4c20-98c5-dae6612267a2\") " Nov 03 00:55:45.059601 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:45.059571 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e7db63-1f77-4c20-98c5-dae6612267a2-kube-api-access-z85qj" pod "59e7db63-1f77-4c20-98c5-dae6612267a2" (UID: "59e7db63-1f77-4c20-98c5-dae6612267a2"). InnerVolumeSpecName "kube-api-access-z85qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:45.159035 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:45.158997 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z85qj\" (UniqueName: \"kubernetes.io/projected/59e7db63-1f77-4c20-98c5-dae6612267a2-kube-api-access-z85qj\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:45.179355 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:45.179321 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e7db63-1f77-4c20-98c5-dae6612267a2" path="/var/lib/kubelet/pods/59e7db63-1f77-4c20-98c5-dae6612267a2/volumes" Nov 03 00:55:45.362630 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:45.362546 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7297-e64e9e"] Nov 03 00:55:45.366854 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:45.366798 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7297-e64e9e"] Nov 03 00:55:45.702573 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:45.702480 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:45.767677 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:45.767639 6400 generic.go:356] "Generic (PLEG): container finished" podID="59e7db63-1f77-4c20-98c5-dae6612267a2" containerID="2ae8d3833e501353be69057ba219f3bb39c3e07e4c689a9b08dedac0ca3f1bb2" exitCode=137 Nov 03 00:55:45.767888 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:45.767684 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-2b3c6a" Nov 03 00:55:45.767888 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:45.767777 6400 scope.go:122] "RemoveContainer" containerID="2ae8d3833e501353be69057ba219f3bb39c3e07e4c689a9b08dedac0ca3f1bb2" Nov 03 00:55:45.768026 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:45.767903 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7297-e64e9e" podUID="4cafd7d0-e316-47e7-a835-ef0e805e2d56" containerName="test-curl-7297-e64e9e" containerID="cri-o://9f9221cbb330aa8bd988b7721f6ca29d7a51403e4fa0843bfadeac34f3a20a9b" gracePeriod=2 Nov 03 00:55:45.777024 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:45.776989 6400 scope.go:122] "RemoveContainer" containerID="2ae8d3833e501353be69057ba219f3bb39c3e07e4c689a9b08dedac0ca3f1bb2" Nov 03 00:55:45.777358 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:45.777337 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae8d3833e501353be69057ba219f3bb39c3e07e4c689a9b08dedac0ca3f1bb2\": container with ID starting with 2ae8d3833e501353be69057ba219f3bb39c3e07e4c689a9b08dedac0ca3f1bb2 not found: ID does not exist" containerID="2ae8d3833e501353be69057ba219f3bb39c3e07e4c689a9b08dedac0ca3f1bb2" Nov 03 00:55:45.777419 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:45.777368 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae8d3833e501353be69057ba219f3bb39c3e07e4c689a9b08dedac0ca3f1bb2"} err="failed to get container status \"2ae8d3833e501353be69057ba219f3bb39c3e07e4c689a9b08dedac0ca3f1bb2\": rpc error: code = NotFound desc = could not find container \"2ae8d3833e501353be69057ba219f3bb39c3e07e4c689a9b08dedac0ca3f1bb2\": container with ID starting with 2ae8d3833e501353be69057ba219f3bb39c3e07e4c689a9b08dedac0ca3f1bb2 not found: ID does not exist" Nov 03 00:55:45.818738 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:45.818687 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-e64e9e\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="4cafd7d0-e316-47e7-a835-ef0e805e2d56" pod="opendatahub/test-curl-7297-e64e9e" Nov 03 00:55:45.869937 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:45.869892 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-e64e9e\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="4cafd7d0-e316-47e7-a835-ef0e805e2d56" pod="opendatahub/test-curl-7297-e64e9e" Nov 03 00:55:46.902187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:46.902143 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" podUID="226105cb-1776-4571-8e64-f0b0fef9bdd6" containerName="main" containerID="cri-o://cde596fa84ace5042e3ae76021da4f6122c4eac97c963ca81103d485d7016802" gracePeriod=45 Nov 03 00:55:47.113536 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.113505 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:47.167511 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.167438 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:55:47.280910 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.280874 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-dshm\") pod \"226105cb-1776-4571-8e64-f0b0fef9bdd6\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " Nov 03 00:55:47.281359 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.281308 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-home\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-home\") pod \"226105cb-1776-4571-8e64-f0b0fef9bdd6\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " Nov 03 00:55:47.281591 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.281572 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-tmp-dir\") pod \"226105cb-1776-4571-8e64-f0b0fef9bdd6\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " Nov 03 00:55:47.281759 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.281733 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-model-cache\") pod \"226105cb-1776-4571-8e64-f0b0fef9bdd6\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " Nov 03 00:55:47.282081 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.281849 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226105cb-1776-4571-8e64-f0b0fef9bdd6-kube-api-access-5vk9p\" (UniqueName: \"kubernetes.io/projected/226105cb-1776-4571-8e64-f0b0fef9bdd6-kube-api-access-5vk9p\") pod \"226105cb-1776-4571-8e64-f0b0fef9bdd6\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " Nov 03 00:55:47.282081 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.281881 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/226105cb-1776-4571-8e64-f0b0fef9bdd6-tls-certs\" (UniqueName: \"kubernetes.io/secret/226105cb-1776-4571-8e64-f0b0fef9bdd6-tls-certs\") pod \"226105cb-1776-4571-8e64-f0b0fef9bdd6\" (UID: \"226105cb-1776-4571-8e64-f0b0fef9bdd6\") " Nov 03 00:55:47.282241 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.281600 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-home" pod "226105cb-1776-4571-8e64-f0b0fef9bdd6" (UID: "226105cb-1776-4571-8e64-f0b0fef9bdd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:55:47.282241 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.281928 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-tmp-dir" pod "226105cb-1776-4571-8e64-f0b0fef9bdd6" (UID: "226105cb-1776-4571-8e64-f0b0fef9bdd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:55:47.282387 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.282236 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-model-cache" pod "226105cb-1776-4571-8e64-f0b0fef9bdd6" (UID: "226105cb-1776-4571-8e64-f0b0fef9bdd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:55:47.282502 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.282479 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:47.282704 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.282509 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:47.282704 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.282523 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:47.284609 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.284577 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-dshm" pod "226105cb-1776-4571-8e64-f0b0fef9bdd6" (UID: "226105cb-1776-4571-8e64-f0b0fef9bdd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:55:47.285704 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.285666 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226105cb-1776-4571-8e64-f0b0fef9bdd6-tls-certs" pod "226105cb-1776-4571-8e64-f0b0fef9bdd6" (UID: "226105cb-1776-4571-8e64-f0b0fef9bdd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:55:47.285878 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.285708 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226105cb-1776-4571-8e64-f0b0fef9bdd6-kube-api-access-5vk9p" pod "226105cb-1776-4571-8e64-f0b0fef9bdd6" (UID: "226105cb-1776-4571-8e64-f0b0fef9bdd6"). InnerVolumeSpecName "kube-api-access-5vk9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:47.332658 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:47.332614 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-e64e9e\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="4cafd7d0-e316-47e7-a835-ef0e805e2d56" pod="opendatahub/test-curl-7297-e64e9e" Nov 03 00:55:47.383363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.383318 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vk9p\" (UniqueName: \"kubernetes.io/projected/226105cb-1776-4571-8e64-f0b0fef9bdd6-kube-api-access-5vk9p\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:47.383363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.383354 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/226105cb-1776-4571-8e64-f0b0fef9bdd6-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:47.383363 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.383365 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/226105cb-1776-4571-8e64-f0b0fef9bdd6-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:47.761685 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.761647 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:47.813548 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.813505 6400 generic.go:356] "Generic (PLEG): container finished" podID="226105cb-1776-4571-8e64-f0b0fef9bdd6" containerID="cde596fa84ace5042e3ae76021da4f6122c4eac97c963ca81103d485d7016802" exitCode=0 Nov 03 00:55:47.813919 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.813801 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" Nov 03 00:55:47.814314 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.813838 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" event={"ID":"226105cb-1776-4571-8e64-f0b0fef9bdd6","Type":"ContainerDied","Data":"cde596fa84ace5042e3ae76021da4f6122c4eac97c963ca81103d485d7016802"} Nov 03 00:55:47.814314 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.814151 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp" event={"ID":"226105cb-1776-4571-8e64-f0b0fef9bdd6","Type":"ContainerDied","Data":"0121b7de0137343f8cc07e4cd2b78153227a17531a9bca4f3f82d523d11da26e"} Nov 03 00:55:47.814314 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.814176 6400 scope.go:122] "RemoveContainer" containerID="cde596fa84ace5042e3ae76021da4f6122c4eac97c963ca81103d485d7016802" Nov 03 00:55:47.866680 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.866563 6400 scope.go:122] "RemoveContainer" containerID="cde596fa84ace5042e3ae76021da4f6122c4eac97c963ca81103d485d7016802" Nov 03 00:55:47.867245 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:47.867171 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde596fa84ace5042e3ae76021da4f6122c4eac97c963ca81103d485d7016802\": container with ID starting with cde596fa84ace5042e3ae76021da4f6122c4eac97c963ca81103d485d7016802 not found: ID does not exist" containerID="cde596fa84ace5042e3ae76021da4f6122c4eac97c963ca81103d485d7016802" Nov 03 00:55:47.867245 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.867208 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde596fa84ace5042e3ae76021da4f6122c4eac97c963ca81103d485d7016802"} err="failed to get container status \"cde596fa84ace5042e3ae76021da4f6122c4eac97c963ca81103d485d7016802\": rpc error: code = NotFound desc = could not find container \"cde596fa84ace5042e3ae76021da4f6122c4eac97c963ca81103d485d7016802\": container with ID starting with cde596fa84ace5042e3ae76021da4f6122c4eac97c963ca81103d485d7016802 not found: ID does not exist" Nov 03 00:55:47.937564 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.937512 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:47.995401 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.995361 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp"] Nov 03 00:55:47.999116 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:47.999090 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b2e795/auth-test-model-b2e795-kserve-7945f8dc57-cptvp"] Nov 03 00:55:48.093865 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:48.091663 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-e64e9e" Nov 03 00:55:48.193479 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:48.193441 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cafd7d0-e316-47e7-a835-ef0e805e2d56-kube-api-access-42rnr\" (UniqueName: \"kubernetes.io/projected/4cafd7d0-e316-47e7-a835-ef0e805e2d56-kube-api-access-42rnr\") pod \"4cafd7d0-e316-47e7-a835-ef0e805e2d56\" (UID: \"4cafd7d0-e316-47e7-a835-ef0e805e2d56\") " Nov 03 00:55:48.195774 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:48.195740 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cafd7d0-e316-47e7-a835-ef0e805e2d56-kube-api-access-42rnr" pod "4cafd7d0-e316-47e7-a835-ef0e805e2d56" (UID: "4cafd7d0-e316-47e7-a835-ef0e805e2d56"). InnerVolumeSpecName "kube-api-access-42rnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:48.295273 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:48.295224 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42rnr\" (UniqueName: \"kubernetes.io/projected/4cafd7d0-e316-47e7-a835-ef0e805e2d56-kube-api-access-42rnr\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:48.820488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:48.820452 6400 generic.go:356] "Generic (PLEG): container finished" podID="4cafd7d0-e316-47e7-a835-ef0e805e2d56" containerID="9f9221cbb330aa8bd988b7721f6ca29d7a51403e4fa0843bfadeac34f3a20a9b" exitCode=137 Nov 03 00:55:48.820679 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:48.820507 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-e64e9e" Nov 03 00:55:48.820679 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:48.820548 6400 scope.go:122] "RemoveContainer" containerID="9f9221cbb330aa8bd988b7721f6ca29d7a51403e4fa0843bfadeac34f3a20a9b" Nov 03 00:55:48.829600 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:48.829577 6400 scope.go:122] "RemoveContainer" containerID="9f9221cbb330aa8bd988b7721f6ca29d7a51403e4fa0843bfadeac34f3a20a9b" Nov 03 00:55:48.829943 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:48.829908 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9221cbb330aa8bd988b7721f6ca29d7a51403e4fa0843bfadeac34f3a20a9b\": container with ID starting with 9f9221cbb330aa8bd988b7721f6ca29d7a51403e4fa0843bfadeac34f3a20a9b not found: ID does not exist" containerID="9f9221cbb330aa8bd988b7721f6ca29d7a51403e4fa0843bfadeac34f3a20a9b" Nov 03 00:55:48.830077 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:48.829951 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9221cbb330aa8bd988b7721f6ca29d7a51403e4fa0843bfadeac34f3a20a9b"} err="failed to get container status \"9f9221cbb330aa8bd988b7721f6ca29d7a51403e4fa0843bfadeac34f3a20a9b\": rpc error: code = NotFound desc = could not find container \"9f9221cbb330aa8bd988b7721f6ca29d7a51403e4fa0843bfadeac34f3a20a9b\": container with ID starting with 9f9221cbb330aa8bd988b7721f6ca29d7a51403e4fa0843bfadeac34f3a20a9b not found: ID does not exist" Nov 03 00:55:48.882751 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:48.882691 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-e64e9e\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="4cafd7d0-e316-47e7-a835-ef0e805e2d56" pod="opendatahub/test-curl-7297-e64e9e" Nov 03 00:55:49.179636 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:49.179563 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="226105cb-1776-4571-8e64-f0b0fef9bdd6" path="/var/lib/kubelet/pods/226105cb-1776-4571-8e64-f0b0fef9bdd6/volumes" Nov 03 00:55:49.180038 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:49.180023 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cafd7d0-e316-47e7-a835-ef0e805e2d56" path="/var/lib/kubelet/pods/4cafd7d0-e316-47e7-a835-ef0e805e2d56/volumes" Nov 03 00:55:50.592911 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:50.592863 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:51.485742 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:51.485708 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:51.663236 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:51.663202 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:51.827538 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:51.827507 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:54.712799 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:54.712767 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:54.887002 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:54.886966 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:55.710314 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:55:55.710278 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:57.153479 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:55:57.153444 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:56:00.456910 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:00.456369 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:05.171488 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:05.171445 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:09.153551 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:09.153513 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:09.800769 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:09.800733 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:13.975057 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:13.975027 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:56:14.227159 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.227080 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh"] Nov 03 00:56:14.227778 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.227756 6400 cpu_manager.go:414] "containerMap: removing container" podUID="226105cb-1776-4571-8e64-f0b0fef9bdd6" containerName="main" Nov 03 00:56:14.227778 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.227779 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226105cb-1776-4571-8e64-f0b0fef9bdd6" containerName="main" Nov 03 00:56:14.228006 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.227802 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4cafd7d0-e316-47e7-a835-ef0e805e2d56" containerName="test-curl-7297-e64e9e" Nov 03 00:56:14.228006 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.227809 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cafd7d0-e316-47e7-a835-ef0e805e2d56" containerName="test-curl-7297-e64e9e" Nov 03 00:56:14.228006 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.227955 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cafd7d0-e316-47e7-a835-ef0e805e2d56" containerName="test-curl-7297-e64e9e" Nov 03 00:56:14.228006 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.227981 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="226105cb-1776-4571-8e64-f0b0fef9bdd6" containerName="main" Nov 03 00:56:14.232768 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.232747 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:14.285789 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.285758 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-f40c6c-serving-cert\"" Nov 03 00:56:14.287780 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.287755 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh"] Nov 03 00:56:14.359476 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.359438 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d55gp\" (UniqueName: \"kubernetes.io/projected/87a3a116-a7eb-4e1e-aea8-113107841624-kube-api-access-d55gp\") pod \"maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:14.359683 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.359566 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:14.359683 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.359670 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:14.359901 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.359872 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a3a116-a7eb-4e1e-aea8-113107841624-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:14.461087 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.461045 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:14.461293 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.461123 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:14.461293 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.461201 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a3a116-a7eb-4e1e-aea8-113107841624-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:14.461293 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.461247 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d55gp\" (UniqueName: \"kubernetes.io/projected/87a3a116-a7eb-4e1e-aea8-113107841624-kube-api-access-d55gp\") pod \"maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:14.461423 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:56:14.461249 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-serving-cert: secret "maas-api-e2e-shared-a-w0-f40c6c-serving-cert" not found Nov 03 00:56:14.461423 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:56:14.461380 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-maas-api-tls podName:87a3a116-a7eb-4e1e-aea8-113107841624 nodeName:}" failed. No retries permitted until 2026-11-03 00:56:14.961360999 +0000 UTC m=+3021.912301153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-maas-api-tls") pod "maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" (UID: "87a3a116-a7eb-4e1e-aea8-113107841624") : secret "maas-api-e2e-shared-a-w0-f40c6c-serving-cert" not found Nov 03 00:56:14.461835 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.461804 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a3a116-a7eb-4e1e-aea8-113107841624-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:14.464043 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.464026 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:14.500557 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.500471 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:14.518099 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.518073 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d55gp\" (UniqueName: \"kubernetes.io/projected/87a3a116-a7eb-4e1e-aea8-113107841624-kube-api-access-d55gp\") pod \"maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:14.662476 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.662442 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:14.968693 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.968652 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:14.971287 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:14.971257 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:15.146867 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:15.146804 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:15.522660 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:15.522628 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh"] Nov 03 00:56:15.526292 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:56:15.526263 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a3a116_a7eb_4e1e_aea8_113107841624.slice/crio-44dc0984b1e26fe3bf1353700c39ed34b41a477d1370f843ba8a6f61a0c1d934 WatchSource:0}: Error finding container 44dc0984b1e26fe3bf1353700c39ed34b41a477d1370f843ba8a6f61a0c1d934: Status 404 returned error can't find the container with id 44dc0984b1e26fe3bf1353700c39ed34b41a477d1370f843ba8a6f61a0c1d934 Nov 03 00:56:15.527791 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:15.527773 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:56:15.618367 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:15.618329 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:15.971000 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:15.970956 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" event={"ID":"87a3a116-a7eb-4e1e-aea8-113107841624","Type":"ContainerStarted","Data":"44dc0984b1e26fe3bf1353700c39ed34b41a477d1370f843ba8a6f61a0c1d934"} Nov 03 00:56:16.979670 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:16.979633 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" event={"ID":"87a3a116-a7eb-4e1e-aea8-113107841624","Type":"ContainerStarted","Data":"90064bb842370308cd7d6abb2679e7004d34351668345305eee7cb0085839a9f"} Nov 03 00:56:16.980093 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:16.979719 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:17.095057 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:17.095008 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" podStartSLOduration=2.140766718 podStartE2EDuration="3.094994562s" totalImagesPullingTime="954.227844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:17.093891191 +0000 UTC m=+3024.044831354" watchObservedRunningTime="2026-11-03 00:56:17.094994562 +0000 UTC m=+3024.045934721" Nov 03 00:56:19.404132 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:19.404098 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:22.989839 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:22.989791 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 00:56:24.128071 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:24.128036 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:24.295494 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:24.295393 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:27.152531 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:56:27.152486 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:56:28.805989 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:28.805950 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:33.505648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:33.505612 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:38.206323 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:38.206286 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:40.509194 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:40.509162 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:42.093398 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:42.093361 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:42.810053 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:42.809903 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:43.049416 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:43.049381 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:43.223910 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:43.223795 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:44.436239 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:44.436203 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7303-532ea2"] Nov 03 00:56:44.439993 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:44.439972 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7303-532ea2" Nov 03 00:56:44.494610 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:44.494587 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7303-532ea2"] Nov 03 00:56:44.541412 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:44.541386 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fz96m\"" Nov 03 00:56:44.552723 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:44.552700 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58n7n\" (UniqueName: \"kubernetes.io/projected/c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec-kube-api-access-58n7n\") pod \"test-curl-7303-532ea2\" (UID: \"c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec\") " pod="opendatahub/test-curl-7303-532ea2" Nov 03 00:56:44.653514 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:44.653483 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58n7n\" (UniqueName: \"kubernetes.io/projected/c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec-kube-api-access-58n7n\") pod \"test-curl-7303-532ea2\" (UID: \"c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec\") " pod="opendatahub/test-curl-7303-532ea2" Nov 03 00:56:44.710294 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:44.710226 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58n7n\" (UniqueName: \"kubernetes.io/projected/c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec-kube-api-access-58n7n\") pod \"test-curl-7303-532ea2\" (UID: \"c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec\") " pod="opendatahub/test-curl-7303-532ea2" Nov 03 00:56:44.753047 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:44.753022 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7303-532ea2" Nov 03 00:56:45.153665 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:45.153637 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7303-532ea2"] Nov 03 00:56:45.155956 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:56:45.155929 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc59b9bcd_49b1_4eaa_8611_d410ccf5c3ec.slice/crio-cf5180469088eaa8c679f25ab4de3346b18b9be6af270a98e5ac391c71d66abd WatchSource:0}: Error finding container cf5180469088eaa8c679f25ab4de3346b18b9be6af270a98e5ac391c71d66abd: Status 404 returned error can't find the container with id cf5180469088eaa8c679f25ab4de3346b18b9be6af270a98e5ac391c71d66abd Nov 03 00:56:46.148322 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:46.148261 6400 generic.go:356] "Generic (PLEG): container finished" podID="e27d8659-f856-403f-a9eb-d86f17f24506" containerID="dc8ed8f15fe481d9f4ce9a6f4a210a69b6629d3b074c5158144f0a49a5df2434" exitCode=0 Nov 03 00:56:46.148773 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:46.148337 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" event={"ID":"e27d8659-f856-403f-a9eb-d86f17f24506","Type":"ContainerDied","Data":"dc8ed8f15fe481d9f4ce9a6f4a210a69b6629d3b074c5158144f0a49a5df2434"} Nov 03 00:56:46.148773 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:46.148746 6400 scope.go:122] "RemoveContainer" containerID="dc8ed8f15fe481d9f4ce9a6f4a210a69b6629d3b074c5158144f0a49a5df2434" Nov 03 00:56:46.149941 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:46.149915 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7303-532ea2" event={"ID":"c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec","Type":"ContainerStarted","Data":"166240e5bdf636366ea0785bf4bfa25d498fe02843b0b5f6d573c356806a92f0"} Nov 03 00:56:46.150056 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:46.149956 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7303-532ea2" event={"ID":"c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec","Type":"ContainerStarted","Data":"cf5180469088eaa8c679f25ab4de3346b18b9be6af270a98e5ac391c71d66abd"} Nov 03 00:56:46.150331 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:46.150307 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:56:46.150383 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:46.150371 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:56:46.150418 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:46.150405 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:56:46.429341 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:46.429294 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7303-532ea2" podStartSLOduration=2.429279456 podStartE2EDuration="2.429279456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:46.429162572 +0000 UTC m=+3053.380102731" watchObservedRunningTime="2026-11-03 00:56:46.429279456 +0000 UTC m=+3053.380219614" Nov 03 00:56:46.805146 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:46.805110 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:47.025181 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:47.025147 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:47.157350 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:47.157265 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" event={"ID":"e27d8659-f856-403f-a9eb-d86f17f24506","Type":"ContainerStarted","Data":"bbd38e8176ef1a958623975369bf98ea38c15ba5dc1089355defecd962c71dd3"} Nov 03 00:56:47.807518 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:47.807488 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:56:47.808751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:47.808714 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 00:56:47.808924 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:47.808771 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 00:56:47.912934 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:47.912902 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7303-532ea2"] Nov 03 00:56:47.916800 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:47.916767 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7303-532ea2"] Nov 03 00:56:48.116259 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:48.116177 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:48.163171 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:48.163111 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7303-532ea2" podUID="c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec" containerName="test-curl-7303-532ea2" containerID="cri-o://166240e5bdf636366ea0785bf4bfa25d498fe02843b0b5f6d573c356806a92f0" gracePeriod=2 Nov 03 00:56:48.213842 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:56:48.213795 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7303-532ea2\" is forbidden: User \"system:node:ip-10-0-135-199.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-199.us-west-1.compute.internal' and this object" podUID="c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec" pod="opendatahub/test-curl-7303-532ea2" Nov 03 00:56:48.807011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:48.806982 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 00:56:48.807285 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:48.807030 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 00:56:49.807409 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:49.807375 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 00:56:49.807804 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:49.807438 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 00:56:50.041314 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:50.041283 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:50.238910 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:50.238801 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:50.464291 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:50.464256 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7303-532ea2" Nov 03 00:56:50.548879 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:50.542362 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec-kube-api-access-58n7n\" (UniqueName: \"kubernetes.io/projected/c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec-kube-api-access-58n7n\") pod \"c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec\" (UID: \"c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec\") " Nov 03 00:56:50.559888 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:50.559800 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec-kube-api-access-58n7n" pod "c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec" (UID: "c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec"). InnerVolumeSpecName "kube-api-access-58n7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:56:50.643653 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:50.643620 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58n7n\" (UniqueName: \"kubernetes.io/projected/c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec-kube-api-access-58n7n\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:50.807012 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:50.806980 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 00:56:50.807210 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:50.807037 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 00:56:51.201272 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:51.201190 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec" path="/var/lib/kubelet/pods/c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec/volumes" Nov 03 00:56:51.211695 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:51.211667 6400 generic.go:356] "Generic (PLEG): container finished" podID="c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec" containerID="166240e5bdf636366ea0785bf4bfa25d498fe02843b0b5f6d573c356806a92f0" exitCode=137 Nov 03 00:56:51.211858 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:51.211713 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7303-532ea2" Nov 03 00:56:51.211858 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:51.211747 6400 scope.go:122] "RemoveContainer" containerID="166240e5bdf636366ea0785bf4bfa25d498fe02843b0b5f6d573c356806a92f0" Nov 03 00:56:51.220281 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:51.220265 6400 scope.go:122] "RemoveContainer" containerID="166240e5bdf636366ea0785bf4bfa25d498fe02843b0b5f6d573c356806a92f0" Nov 03 00:56:51.220552 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:56:51.220534 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"166240e5bdf636366ea0785bf4bfa25d498fe02843b0b5f6d573c356806a92f0\": container with ID starting with 166240e5bdf636366ea0785bf4bfa25d498fe02843b0b5f6d573c356806a92f0 not found: ID does not exist" containerID="166240e5bdf636366ea0785bf4bfa25d498fe02843b0b5f6d573c356806a92f0" Nov 03 00:56:51.220589 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:51.220562 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166240e5bdf636366ea0785bf4bfa25d498fe02843b0b5f6d573c356806a92f0"} err="failed to get container status \"166240e5bdf636366ea0785bf4bfa25d498fe02843b0b5f6d573c356806a92f0\": rpc error: code = NotFound desc = could not find container \"166240e5bdf636366ea0785bf4bfa25d498fe02843b0b5f6d573c356806a92f0\": container with ID starting with 166240e5bdf636366ea0785bf4bfa25d498fe02843b0b5f6d573c356806a92f0 not found: ID does not exist" Nov 03 00:56:51.539267 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:51.539240 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:56:51.807953 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:51.807867 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 00:56:51.807953 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:51.807922 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 00:56:52.812787 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:52.812752 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:56:52.813999 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:52.813980 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 00:56:53.128489 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:53.128395 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:57.154053 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:56:57.154017 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:56:57.804657 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:56:57.804619 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:02.398058 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:02.397967 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:02.585849 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:02.584845 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:02.770971 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:02.770895 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:07.244011 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:07.243981 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:11.762698 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:11.762662 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:57:11.908124 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:11.908086 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:14.467648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:14.467618 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:15.960039 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:15.960001 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:16.617648 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:16.617608 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:21.401566 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:21.401525 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:26.101077 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:26.101036 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:27.152312 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:57:27.152265 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:57:30.805150 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:30.805112 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:32.677513 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:32.677475 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:35.559257 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:35.559215 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:40.455941 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:40.455902 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:41.120213 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:41.120174 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:41.567931 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:41.567893 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:42.131598 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.131558 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj"] Nov 03 00:57:42.132140 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.132121 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec" containerName="test-curl-7303-532ea2" Nov 03 00:57:42.132250 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.132141 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec" containerName="test-curl-7303-532ea2" Nov 03 00:57:42.132342 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.132328 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59b9bcd-49b1-4eaa-8611-d410ccf5c3ec" containerName="test-curl-7303-532ea2" Nov 03 00:57:42.135966 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.135943 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.188519 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.188487 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-3b1d36-openshift-default-dockercfg-nqkhm\"" Nov 03 00:57:42.224457 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.224381 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-socket\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.224457 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.224458 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-certs\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.224721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.224489 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-data\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.224721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.224525 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfm6z\" (UniqueName: \"kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-kube-api-access-tfm6z\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.224721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.224595 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-credential-socket\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.224721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.224625 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e7ca86-4542-4352-889a-bc6181318baa-istio-podinfo\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.224721 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.224656 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e7ca86-4542-4352-889a-bc6181318baa-istiod-ca-cert\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.225094 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.224969 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-istio-token\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.225094 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.225032 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-envoy\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.253394 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.253359 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj"] Nov 03 00:57:42.320867 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.320829 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:42.326063 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.326037 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e7ca86-4542-4352-889a-bc6181318baa-istio-podinfo\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.326207 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.326091 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e7ca86-4542-4352-889a-bc6181318baa-istiod-ca-cert\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.326304 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.326281 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-istio-token\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.326367 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.326319 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-envoy\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.326367 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.326360 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-socket\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.326479 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.326392 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-certs\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.326479 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.326424 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-data\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.326479 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.326456 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfm6z\" (UniqueName: \"kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-kube-api-access-tfm6z\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.326654 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.326532 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-credential-socket\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.327066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.326880 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-certs\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.327066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.326909 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-data\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.327066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.326977 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-socket\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.327066 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.327025 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e7ca86-4542-4352-889a-bc6181318baa-istiod-ca-cert\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.327261 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.327135 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-credential-socket\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.329387 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.329361 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-envoy\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.329493 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.329373 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e7ca86-4542-4352-889a-bc6181318baa-istio-podinfo\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.383891 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.383787 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfm6z\" (UniqueName: \"kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-kube-api-access-tfm6z\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.383891 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.383793 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-istio-token\") pod \"e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.449397 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.449359 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:42.659795 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.659697 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:42.879617 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.879588 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj"] Nov 03 00:57:42.882161 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:57:42.882120 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e7ca86_4542_4352_889a_bc6181318baa.slice/crio-3f4954f6a3bae63eb56b7211736b61c4bd391f2c67a3f2a02a3f79a20c80aad0 WatchSource:0}: Error finding container 3f4954f6a3bae63eb56b7211736b61c4bd391f2c67a3f2a02a3f79a20c80aad0: Status 404 returned error can't find the container with id 3f4954f6a3bae63eb56b7211736b61c4bd391f2c67a3f2a02a3f79a20c80aad0 Nov 03 00:57:42.885074 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.885034 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:57:42.885148 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.885125 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:57:42.885187 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:42.885168 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:57:43.394201 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:43.394159 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:43.538560 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:43.538517 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" event={"ID":"73e7ca86-4542-4352-889a-bc6181318baa","Type":"ContainerStarted","Data":"d00cb8c36ec37c3d173e7ef91071d3e1a522e2881fd40efdee111185f7aa5ba1"} Nov 03 00:57:43.538751 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:43.538588 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" event={"ID":"73e7ca86-4542-4352-889a-bc6181318baa","Type":"ContainerStarted","Data":"3f4954f6a3bae63eb56b7211736b61c4bd391f2c67a3f2a02a3f79a20c80aad0"} Nov 03 00:57:43.657024 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:43.656917 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" podStartSLOduration=1.65690228 podStartE2EDuration="1.65690228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:43.655564063 +0000 UTC m=+3110.606504233" watchObservedRunningTime="2026-11-03 00:57:43.65690228 +0000 UTC m=+3110.607842438" Nov 03 00:57:43.990946 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:43.990815 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:44.449546 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:44.449515 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:44.457800 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:44.457769 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:44.558845 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:44.558622 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:44.570099 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:44.570067 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:57:45.284120 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:45.284089 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:46.050636 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:46.050605 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:46.257205 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:46.257167 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:46.535638 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:46.535604 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:46.715719 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:46.715685 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:46.906434 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:46.906350 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:49.801787 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:49.801738 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:50.017462 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:50.017431 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:52.567650 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:52.567577 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:52.739614 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:52.739577 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:54.968999 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:54.968963 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:55.211668 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:55.211638 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:55.616560 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:57:55.616524 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:57.152015 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:57:57.151973 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:58:00.259165 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:00.259119 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:04.805207 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:04.805171 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:04.995844 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:04.993334 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:05.162782 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:05.162691 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:05.535878 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:05.535840 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:05.812401 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:05.812328 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:09.813069 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:09.813040 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:10.221130 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:10.221058 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:10.397617 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:10.397576 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:10.558292 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:10.558261 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:13.452237 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:13.452208 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:13.628930 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:13.628877 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:14.952959 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:14.952920 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:16.320617 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:16.320580 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:19.653939 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:19.653903 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:23.598935 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:23.598900 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:58:24.322342 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:24.322309 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:27.153510 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:58:27.153469 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:58:29.124033 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:29.123990 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:30.142603 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:30.142539 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:31.507732 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:31.507654 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:31.826800 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:31.826770 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:32.001783 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:31.999667 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx"] Nov 03 00:58:32.004212 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.004181 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.084674 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.084584 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx"] Nov 03 00:58:32.093908 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.093862 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-envoy\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.094182 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.094158 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-podinfo\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.094263 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.094202 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-credential-socket\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.094263 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.094230 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-data\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.094368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.094286 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rchwg\" (UniqueName: \"kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-kube-api-access-rchwg\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.094368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.094322 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdae0d76-2372-4d89-89e2-84e69f787b99-istiod-ca-cert\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.094525 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.094499 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-socket\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.094578 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.094561 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-token\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.094639 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.094625 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-certs\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.116857 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.116804 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:32.125796 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.125768 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-642e6a-openshift-default-dockercfg-zhddv\"" Nov 03 00:58:32.196430 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.196390 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-envoy\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.196868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.196493 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-podinfo\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.196868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.196534 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-credential-socket\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.196868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.196559 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-data\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.196868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.196624 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rchwg\" (UniqueName: \"kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-kube-api-access-rchwg\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.196868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.196660 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdae0d76-2372-4d89-89e2-84e69f787b99-istiod-ca-cert\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.196868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.196691 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-socket\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.196868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.196732 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-token\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.196868 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.196805 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-certs\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.197445 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.197077 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-credential-socket\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.197754 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.197728 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdae0d76-2372-4d89-89e2-84e69f787b99-istiod-ca-cert\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.197985 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.197958 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-data\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.198065 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.198011 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-socket\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.198109 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.198061 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-certs\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.200351 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.200328 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-podinfo\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.200447 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.200334 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-envoy\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.271002 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.270962 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rchwg\" (UniqueName: \"kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-kube-api-access-rchwg\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.286908 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.286875 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-token\") pod \"e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.318944 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.318909 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:32.320670 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.320638 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:32.760387 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.760356 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx"] Nov 03 00:58:32.762676 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:58:32.762644 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdae0d76_2372_4d89_89e2_84e69f787b99.slice/crio-fc930b2689b6719a80bc0eb5c9085f21ff05c08b9819ae59b645e1f1578bc546 WatchSource:0}: Error finding container fc930b2689b6719a80bc0eb5c9085f21ff05c08b9819ae59b645e1f1578bc546: Status 404 returned error can't find the container with id fc930b2689b6719a80bc0eb5c9085f21ff05c08b9819ae59b645e1f1578bc546 Nov 03 00:58:32.765505 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.765468 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:58:32.765622 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.765554 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:58:32.765622 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.765600 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:58:32.845937 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:32.845907 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" event={"ID":"bdae0d76-2372-4d89-89e2-84e69f787b99","Type":"ContainerStarted","Data":"fc930b2689b6719a80bc0eb5c9085f21ff05c08b9819ae59b645e1f1578bc546"} Nov 03 00:58:33.852777 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:33.852719 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" event={"ID":"bdae0d76-2372-4d89-89e2-84e69f787b99","Type":"ContainerStarted","Data":"df6735d6ecd70913368ee05b400ad16ec3eed9791ea809484bce14479d17b092"} Nov 03 00:58:34.019961 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:34.019904 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" podStartSLOduration=3.019884753 podStartE2EDuration="3.019884753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:34.016225325 +0000 UTC m=+3160.967165483" watchObservedRunningTime="2026-11-03 00:58:34.019884753 +0000 UTC m=+3160.970824911" Nov 03 00:58:34.264736 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:34.263402 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:34.321458 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:34.321427 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:34.327436 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:34.327414 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:34.861048 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:34.861003 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:34.862595 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:34.862479 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 00:58:36.869881 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:36.869832 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:37.046572 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:37.046535 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:37.218431 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:37.218349 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:39.216572 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:39.216535 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:40.119057 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:40.119025 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:40.307901 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:40.307867 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:44.350680 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:44.350637 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:49.334953 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:49.334920 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:54.027290 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:54.027245 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:54.512610 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:54.512578 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:58:57.152217 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:58:57.152183 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:58:58.708246 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:58:58.708189 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:03.494254 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:03.494220 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:06.849919 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:06.849883 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:07.708003 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:07.707969 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752"] Nov 03 00:59:07.712092 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:07.712072 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:07.815015 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:07.814981 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-3b1d36-serving-cert\"" Nov 03 00:59:07.817099 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:07.817079 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752"] Nov 03 00:59:07.906339 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:07.906312 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:07.906771 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:07.906361 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e8b788-c97d-4739-821a-de697c19d429-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:07.906771 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:07.906587 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:07.906771 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:07.906644 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccb6b\" (UniqueName: \"kubernetes.io/projected/c2e8b788-c97d-4739-821a-de697c19d429-kube-api-access-ccb6b\") pod \"maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:07.922038 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:07.922010 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:08.008307 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:08.008230 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:08.008307 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:08.008280 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccb6b\" (UniqueName: \"kubernetes.io/projected/c2e8b788-c97d-4739-821a-de697c19d429-kube-api-access-ccb6b\") pod \"maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:08.008551 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:08.008316 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:08.008551 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:08.008345 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e8b788-c97d-4739-821a-de697c19d429-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:08.008551 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:59:08.008361 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-serving-cert: secret "maas-api-e2e-shared-b-w0-3b1d36-serving-cert" not found Nov 03 00:59:08.008551 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:59:08.008416 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-maas-api-tls podName:c2e8b788-c97d-4739-821a-de697c19d429 nodeName:}" failed. No retries permitted until 2026-11-03 00:59:08.508400356 +0000 UTC m=+3195.459340492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-maas-api-tls") pod "maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" (UID: "c2e8b788-c97d-4739-821a-de697c19d429") : secret "maas-api-e2e-shared-b-w0-3b1d36-serving-cert" not found Nov 03 00:59:08.009079 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:08.009062 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e8b788-c97d-4739-821a-de697c19d429-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:08.011001 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:08.010980 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:08.065075 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:08.065044 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccb6b\" (UniqueName: \"kubernetes.io/projected/c2e8b788-c97d-4739-821a-de697c19d429-kube-api-access-ccb6b\") pod \"maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:08.130126 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:08.130104 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:08.296432 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:08.296403 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:08.513366 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:08.513330 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:08.515614 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:08.515592 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:08.624858 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:08.624773 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:08.994754 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:08.994728 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752"] Nov 03 00:59:08.996515 ip-10-0-135-199 kubenswrapper[6400]: W1103 00:59:08.996487 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e8b788_c97d_4739_821a_de697c19d429.slice/crio-e4cfd74ac29ac0268dd748d8470880e59e72c73d5f0162fcc1f87bcd32e087de WatchSource:0}: Error finding container e4cfd74ac29ac0268dd748d8470880e59e72c73d5f0162fcc1f87bcd32e087de: Status 404 returned error can't find the container with id e4cfd74ac29ac0268dd748d8470880e59e72c73d5f0162fcc1f87bcd32e087de Nov 03 00:59:09.041734 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:09.041707 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" event={"ID":"c2e8b788-c97d-4739-821a-de697c19d429","Type":"ContainerStarted","Data":"e4cfd74ac29ac0268dd748d8470880e59e72c73d5f0162fcc1f87bcd32e087de"} Nov 03 00:59:10.048838 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:10.048784 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" event={"ID":"c2e8b788-c97d-4739-821a-de697c19d429","Type":"ContainerStarted","Data":"2453ec56d4338dad95df746195755a7552c453fe078609c659d8fecd31113ea2"} Nov 03 00:59:10.049198 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:10.048869 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:10.163512 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:10.163461 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" podStartSLOduration=2.2391101 podStartE2EDuration="3.163447278s" totalImagesPullingTime="924.337178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:10.162496046 +0000 UTC m=+3197.113436215" watchObservedRunningTime="2026-11-03 00:59:10.163447278 +0000 UTC m=+3197.114387429" Nov 03 00:59:12.888809 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:12.888773 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:16.059145 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:16.059120 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 00:59:16.640534 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:16.640490 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:17.444089 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:17.444053 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:19.716130 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:19.716096 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:22.334364 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:22.334327 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:27.023323 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:27.023285 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:27.151292 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:59:27.151254 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:59:31.716836 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:31.716784 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:33.202153 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:33.202105 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:33.408920 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:33.408885 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:33.597933 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:33.597903 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 00:59:36.473338 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:36.473244 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:41.161058 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:41.161017 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:41.388765 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:41.388519 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:41.540973 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:41.540932 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:44.251259 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:44.251225 6400 generic.go:356] "Generic (PLEG): container finished" podID="73e7ca86-4542-4352-889a-bc6181318baa" containerID="d00cb8c36ec37c3d173e7ef91071d3e1a522e2881fd40efdee111185f7aa5ba1" exitCode=0 Nov 03 00:59:44.251690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:44.251265 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" event={"ID":"73e7ca86-4542-4352-889a-bc6181318baa","Type":"ContainerDied","Data":"d00cb8c36ec37c3d173e7ef91071d3e1a522e2881fd40efdee111185f7aa5ba1"} Nov 03 00:59:44.251690 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:44.251656 6400 scope.go:122] "RemoveContainer" containerID="d00cb8c36ec37c3d173e7ef91071d3e1a522e2881fd40efdee111185f7aa5ba1" Nov 03 00:59:44.253300 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:44.253269 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:59:44.253362 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:44.253344 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:59:44.253401 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:44.253377 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 00:59:44.559417 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:44.559384 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:59:45.257753 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:45.257721 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" event={"ID":"73e7ca86-4542-4352-889a-bc6181318baa","Type":"ContainerStarted","Data":"7e334ae080fa5056225586578d2b4f46a65de809a3ab78f40003c6256c082f48"} Nov 03 00:59:45.449879 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:45.449840 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:59:45.451167 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:45.451134 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 00:59:45.451330 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:45.451228 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 00:59:45.874400 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:45.874366 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:46.407310 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:46.407275 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:46.449890 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:46.449854 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 00:59:46.450130 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:46.449923 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 00:59:47.450693 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:47.450653 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 00:59:47.451298 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:47.450716 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 00:59:48.449680 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:48.449636 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 00:59:48.449920 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:48.449688 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 00:59:49.450368 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:49.450332 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 00:59:49.450906 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:49.450395 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 00:59:50.449722 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:50.449681 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 00:59:50.449950 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:50.449756 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 00:59:51.160401 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:51.160373 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:51.455862 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:51.455775 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:59:51.456235 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:51.456210 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:59:51.456813 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:51.456795 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 00:59:51.974297 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:51.974263 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:52.765434 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:52.765403 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:52.939021 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:52.938428 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:55.166426 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:55.166394 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:55.364997 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:55.364961 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:55.631163 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:55.631132 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:56.104299 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:56.104270 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:57.151918 ip-10-0-135-199 kubenswrapper[6400]: E1103 00:59:57.151884 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 00:59:57.653625 ip-10-0-135-199 kubenswrapper[6400]: I1103 00:59:57.653582 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:00.211690 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.211658 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2"] Nov 03 01:00:00.222786 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.222731 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" Nov 03 01:00:00.234704 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.234683 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f017544e-4cbe-4f21-abdc-70b6c6ab8981-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2\" (UID: \"f017544e-4cbe-4f21-abdc-70b6c6ab8981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" Nov 03 01:00:00.234844 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.234810 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbq9x\" (UniqueName: \"kubernetes.io/projected/f017544e-4cbe-4f21-abdc-70b6c6ab8981-kube-api-access-wbq9x\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2\" (UID: \"f017544e-4cbe-4f21-abdc-70b6c6ab8981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" Nov 03 01:00:00.240465 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.240444 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w"] Nov 03 01:00:00.244193 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.244175 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" Nov 03 01:00:00.257008 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.256979 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4"] Nov 03 01:00:00.260399 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.260382 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" Nov 03 01:00:00.260888 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.260868 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2"] Nov 03 01:00:00.264236 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.264218 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" Nov 03 01:00:00.290141 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.290117 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x"] Nov 03 01:00:00.293648 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.293632 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" Nov 03 01:00:00.296445 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.296424 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm"] Nov 03 01:00:00.299953 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.299935 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" Nov 03 01:00:00.330751 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.330724 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w"] Nov 03 01:00:00.330977 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.330957 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2"] Nov 03 01:00:00.335275 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.335250 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:00:00.335518 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.335492 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lz7\" (UniqueName: \"kubernetes.io/projected/780105c6-b952-4193-b287-9e93040ea9e6-kube-api-access-k9lz7\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x\" (UID: \"780105c6-b952-4193-b287-9e93040ea9e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" Nov 03 01:00:00.335688 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.335670 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xnx9\" (UniqueName: \"kubernetes.io/projected/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-kube-api-access-7xnx9\") pod \"maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w\" (UID: \"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" Nov 03 01:00:00.335805 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.335698 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trtf7\" (UniqueName: \"kubernetes.io/projected/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-kube-api-access-trtf7\") pod \"maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm\" (UID: \"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" Nov 03 01:00:00.335922 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.335877 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w\" (UID: \"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" Nov 03 01:00:00.335922 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.335914 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbq9x\" (UniqueName: \"kubernetes.io/projected/f017544e-4cbe-4f21-abdc-70b6c6ab8981-kube-api-access-wbq9x\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2\" (UID: \"f017544e-4cbe-4f21-abdc-70b6c6ab8981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" Nov 03 01:00:00.336040 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.335975 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780105c6-b952-4193-b287-9e93040ea9e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x\" (UID: \"780105c6-b952-4193-b287-9e93040ea9e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" Nov 03 01:00:00.336097 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.336041 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2\" (UID: \"0f55000e-0fcb-4045-a07b-7f89fe9f20c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" Nov 03 01:00:00.336097 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.336064 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tknjf\" (UniqueName: \"kubernetes.io/projected/6da8cdf3-0d59-48ca-ad28-6a85adac1544-kube-api-access-tknjf\") pod \"maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4\" (UID: \"6da8cdf3-0d59-48ca-ad28-6a85adac1544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" Nov 03 01:00:00.336300 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.336261 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f017544e-4cbe-4f21-abdc-70b6c6ab8981-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2\" (UID: \"f017544e-4cbe-4f21-abdc-70b6c6ab8981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" Nov 03 01:00:00.336394 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.336319 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm\" (UID: \"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" Nov 03 01:00:00.336458 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.336395 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da8cdf3-0d59-48ca-ad28-6a85adac1544-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4\" (UID: \"6da8cdf3-0d59-48ca-ad28-6a85adac1544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" Nov 03 01:00:00.336458 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.336430 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krnsn\" (UniqueName: \"kubernetes.io/projected/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-kube-api-access-krnsn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2\" (UID: \"0f55000e-0fcb-4045-a07b-7f89fe9f20c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" Nov 03 01:00:00.336917 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.336897 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f017544e-4cbe-4f21-abdc-70b6c6ab8981-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2\" (UID: \"f017544e-4cbe-4f21-abdc-70b6c6ab8981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" Nov 03 01:00:00.368407 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.368372 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4"] Nov 03 01:00:00.380118 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.380083 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x"] Nov 03 01:00:00.382752 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.382727 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm"] Nov 03 01:00:00.399566 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.397002 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2"] Nov 03 01:00:00.403182 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.401601 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbq9x\" (UniqueName: \"kubernetes.io/projected/f017544e-4cbe-4f21-abdc-70b6c6ab8981-kube-api-access-wbq9x\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2\" (UID: \"f017544e-4cbe-4f21-abdc-70b6c6ab8981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" Nov 03 01:00:00.437549 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.437518 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da8cdf3-0d59-48ca-ad28-6a85adac1544-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4\" (UID: \"6da8cdf3-0d59-48ca-ad28-6a85adac1544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" Nov 03 01:00:00.437690 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.437557 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krnsn\" (UniqueName: \"kubernetes.io/projected/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-kube-api-access-krnsn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2\" (UID: \"0f55000e-0fcb-4045-a07b-7f89fe9f20c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" Nov 03 01:00:00.437690 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.437589 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lz7\" (UniqueName: \"kubernetes.io/projected/780105c6-b952-4193-b287-9e93040ea9e6-kube-api-access-k9lz7\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x\" (UID: \"780105c6-b952-4193-b287-9e93040ea9e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" Nov 03 01:00:00.437690 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.437614 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xnx9\" (UniqueName: \"kubernetes.io/projected/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-kube-api-access-7xnx9\") pod \"maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w\" (UID: \"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" Nov 03 01:00:00.437690 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.437634 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trtf7\" (UniqueName: \"kubernetes.io/projected/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-kube-api-access-trtf7\") pod \"maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm\" (UID: \"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" Nov 03 01:00:00.437943 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.437690 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w\" (UID: \"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" Nov 03 01:00:00.437943 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.437739 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780105c6-b952-4193-b287-9e93040ea9e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x\" (UID: \"780105c6-b952-4193-b287-9e93040ea9e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" Nov 03 01:00:00.437943 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.437812 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2\" (UID: \"0f55000e-0fcb-4045-a07b-7f89fe9f20c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" Nov 03 01:00:00.437943 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.437873 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknjf\" (UniqueName: \"kubernetes.io/projected/6da8cdf3-0d59-48ca-ad28-6a85adac1544-kube-api-access-tknjf\") pod \"maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4\" (UID: \"6da8cdf3-0d59-48ca-ad28-6a85adac1544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" Nov 03 01:00:00.437943 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.437902 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm\" (UID: \"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" Nov 03 01:00:00.438359 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.438339 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da8cdf3-0d59-48ca-ad28-6a85adac1544-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4\" (UID: \"6da8cdf3-0d59-48ca-ad28-6a85adac1544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" Nov 03 01:00:00.438472 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.438452 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2\" (UID: \"0f55000e-0fcb-4045-a07b-7f89fe9f20c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" Nov 03 01:00:00.438615 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.438597 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm\" (UID: \"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" Nov 03 01:00:00.438675 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.438601 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780105c6-b952-4193-b287-9e93040ea9e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x\" (UID: \"780105c6-b952-4193-b287-9e93040ea9e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" Nov 03 01:00:00.438713 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.438680 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w\" (UID: \"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" Nov 03 01:00:00.494322 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.494257 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lz7\" (UniqueName: \"kubernetes.io/projected/780105c6-b952-4193-b287-9e93040ea9e6-kube-api-access-k9lz7\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x\" (UID: \"780105c6-b952-4193-b287-9e93040ea9e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" Nov 03 01:00:00.494578 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.494550 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krnsn\" (UniqueName: \"kubernetes.io/projected/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-kube-api-access-krnsn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2\" (UID: \"0f55000e-0fcb-4045-a07b-7f89fe9f20c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" Nov 03 01:00:00.495109 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.495086 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tknjf\" (UniqueName: \"kubernetes.io/projected/6da8cdf3-0d59-48ca-ad28-6a85adac1544-kube-api-access-tknjf\") pod \"maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4\" (UID: \"6da8cdf3-0d59-48ca-ad28-6a85adac1544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" Nov 03 01:00:00.495940 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.495923 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trtf7\" (UniqueName: \"kubernetes.io/projected/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-kube-api-access-trtf7\") pod \"maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm\" (UID: \"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" Nov 03 01:00:00.496171 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.496154 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xnx9\" (UniqueName: \"kubernetes.io/projected/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-kube-api-access-7xnx9\") pod \"maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w\" (UID: \"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" Nov 03 01:00:00.537024 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.537002 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" Nov 03 01:00:00.556111 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.555942 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" Nov 03 01:00:00.573303 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.573275 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" Nov 03 01:00:00.579317 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.579291 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" Nov 03 01:00:00.615854 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.609261 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" Nov 03 01:00:00.618756 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.618094 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" Nov 03 01:00:00.936503 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.936472 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2"] Nov 03 01:00:00.937740 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:00:00.937718 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf017544e_4cbe_4f21_abdc_70b6c6ab8981.slice/crio-84367fda87f8c95e7a2e60af70297c1d2e20ae3bfb3732dd6fead39812722468 WatchSource:0}: Error finding container 84367fda87f8c95e7a2e60af70297c1d2e20ae3bfb3732dd6fead39812722468: Status 404 returned error can't find the container with id 84367fda87f8c95e7a2e60af70297c1d2e20ae3bfb3732dd6fead39812722468 Nov 03 01:00:00.951266 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.951240 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:00.975122 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:00.975095 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w"] Nov 03 01:00:00.976097 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:00:00.976046 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f2f5d3_8ed3_417f_b35f_81e082b77aa2.slice/crio-240673cad6d8e4e1ee39b60965fdd547c29372991c0bc387e1ea352987924dee WatchSource:0}: Error finding container 240673cad6d8e4e1ee39b60965fdd547c29372991c0bc387e1ea352987924dee: Status 404 returned error can't find the container with id 240673cad6d8e4e1ee39b60965fdd547c29372991c0bc387e1ea352987924dee Nov 03 01:00:01.018176 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:01.018154 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2"] Nov 03 01:00:01.019563 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:00:01.019545 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f55000e_0fcb_4045_a07b_7f89fe9f20c3.slice/crio-00b39f1c5b59c381e1f7f1a11bd8afa580221635cc9f7dc42a71e5338db9068e WatchSource:0}: Error finding container 00b39f1c5b59c381e1f7f1a11bd8afa580221635cc9f7dc42a71e5338db9068e: Status 404 returned error can't find the container with id 00b39f1c5b59c381e1f7f1a11bd8afa580221635cc9f7dc42a71e5338db9068e Nov 03 01:00:01.024560 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:01.024536 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4"] Nov 03 01:00:01.026017 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:00:01.025998 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da8cdf3_0d59_48ca_ad28_6a85adac1544.slice/crio-172429d96373e23cd5a53385ae8d61b7f1c524cb37803a590e4dbba77d22baec WatchSource:0}: Error finding container 172429d96373e23cd5a53385ae8d61b7f1c524cb37803a590e4dbba77d22baec: Status 404 returned error can't find the container with id 172429d96373e23cd5a53385ae8d61b7f1c524cb37803a590e4dbba77d22baec Nov 03 01:00:01.055037 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:01.052546 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x"] Nov 03 01:00:01.055037 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:01.053807 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm"] Nov 03 01:00:01.055037 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:00:01.053978 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780105c6_b952_4193_b287_9e93040ea9e6.slice/crio-a6d3598c1a524f7782a580f0b8a227c61abf52276f09f8d3bb6bb927b7e479f6 WatchSource:0}: Error finding container a6d3598c1a524f7782a580f0b8a227c61abf52276f09f8d3bb6bb927b7e479f6: Status 404 returned error can't find the container with id a6d3598c1a524f7782a580f0b8a227c61abf52276f09f8d3bb6bb927b7e479f6 Nov 03 01:00:01.055037 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:00:01.054267 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f647fa6_cfa8_4cb7_b8dd_046eb07dcf5c.slice/crio-3886560a5266ebc5446dbd1b639b0c249fd71cb93bdce8226693a4d1610660b5 WatchSource:0}: Error finding container 3886560a5266ebc5446dbd1b639b0c249fd71cb93bdce8226693a4d1610660b5: Status 404 returned error can't find the container with id 3886560a5266ebc5446dbd1b639b0c249fd71cb93bdce8226693a4d1610660b5 Nov 03 01:00:01.379039 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:01.379004 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" event={"ID":"780105c6-b952-4193-b287-9e93040ea9e6","Type":"ContainerStarted","Data":"a6d3598c1a524f7782a580f0b8a227c61abf52276f09f8d3bb6bb927b7e479f6"} Nov 03 01:00:01.380431 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:01.380395 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" event={"ID":"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c","Type":"ContainerStarted","Data":"3886560a5266ebc5446dbd1b639b0c249fd71cb93bdce8226693a4d1610660b5"} Nov 03 01:00:01.381983 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:01.381953 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" event={"ID":"6da8cdf3-0d59-48ca-ad28-6a85adac1544","Type":"ContainerStarted","Data":"172429d96373e23cd5a53385ae8d61b7f1c524cb37803a590e4dbba77d22baec"} Nov 03 01:00:01.383347 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:01.383325 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" event={"ID":"0f55000e-0fcb-4045-a07b-7f89fe9f20c3","Type":"ContainerStarted","Data":"00b39f1c5b59c381e1f7f1a11bd8afa580221635cc9f7dc42a71e5338db9068e"} Nov 03 01:00:01.384633 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:01.384611 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" event={"ID":"f017544e-4cbe-4f21-abdc-70b6c6ab8981","Type":"ContainerStarted","Data":"84367fda87f8c95e7a2e60af70297c1d2e20ae3bfb3732dd6fead39812722468"} Nov 03 01:00:01.386298 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:01.386279 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" event={"ID":"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2","Type":"ContainerStarted","Data":"240673cad6d8e4e1ee39b60965fdd547c29372991c0bc387e1ea352987924dee"} Nov 03 01:00:02.393835 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.393783 6400 generic.go:356] "Generic (PLEG): container finished" podID="780105c6-b952-4193-b287-9e93040ea9e6" containerID="22b04e53d468f3f2aa00709b92da5d089717830f126053c043347f26957c63c9" exitCode=60 Nov 03 01:00:02.394237 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.393869 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" event={"ID":"780105c6-b952-4193-b287-9e93040ea9e6","Type":"ContainerDied","Data":"22b04e53d468f3f2aa00709b92da5d089717830f126053c043347f26957c63c9"} Nov 03 01:00:02.394237 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.394139 6400 scope.go:122] "RemoveContainer" containerID="22b04e53d468f3f2aa00709b92da5d089717830f126053c043347f26957c63c9" Nov 03 01:00:02.395536 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.395501 6400 generic.go:356] "Generic (PLEG): container finished" podID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" containerID="4699dbd6c28d2c2ae4332cb4c6cbdea2816d97357f7cb77f4860864af1ae8684" exitCode=60 Nov 03 01:00:02.395536 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.395525 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" event={"ID":"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c","Type":"ContainerDied","Data":"4699dbd6c28d2c2ae4332cb4c6cbdea2816d97357f7cb77f4860864af1ae8684"} Nov 03 01:00:02.395768 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.395754 6400 scope.go:122] "RemoveContainer" containerID="4699dbd6c28d2c2ae4332cb4c6cbdea2816d97357f7cb77f4860864af1ae8684" Nov 03 01:00:02.398147 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.398118 6400 generic.go:356] "Generic (PLEG): container finished" podID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" containerID="d445c9f7e18f7acf4a7b289966c1c45dbe246fdcff320f6a75aadd72bfbe7d7a" exitCode=60 Nov 03 01:00:02.398337 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.398306 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" event={"ID":"6da8cdf3-0d59-48ca-ad28-6a85adac1544","Type":"ContainerDied","Data":"d445c9f7e18f7acf4a7b289966c1c45dbe246fdcff320f6a75aadd72bfbe7d7a"} Nov 03 01:00:02.398652 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.398622 6400 scope.go:122] "RemoveContainer" containerID="d445c9f7e18f7acf4a7b289966c1c45dbe246fdcff320f6a75aadd72bfbe7d7a" Nov 03 01:00:02.410042 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.410014 6400 generic.go:356] "Generic (PLEG): container finished" podID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" containerID="a07eae6946e7d75effb755339de55f7ec24a508548b71becb9b5f6a82df6e3f6" exitCode=60 Nov 03 01:00:02.410124 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.410104 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" event={"ID":"0f55000e-0fcb-4045-a07b-7f89fe9f20c3","Type":"ContainerDied","Data":"a07eae6946e7d75effb755339de55f7ec24a508548b71becb9b5f6a82df6e3f6"} Nov 03 01:00:02.410356 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.410337 6400 scope.go:122] "RemoveContainer" containerID="a07eae6946e7d75effb755339de55f7ec24a508548b71becb9b5f6a82df6e3f6" Nov 03 01:00:02.412645 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.412622 6400 generic.go:356] "Generic (PLEG): container finished" podID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" containerID="7e83f3b2e46a5a4f5cd47b7df1c455c5f522f6fa11427d9cdc4a0b0759f3d301" exitCode=60 Nov 03 01:00:02.412724 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.412683 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" event={"ID":"f017544e-4cbe-4f21-abdc-70b6c6ab8981","Type":"ContainerDied","Data":"7e83f3b2e46a5a4f5cd47b7df1c455c5f522f6fa11427d9cdc4a0b0759f3d301"} Nov 03 01:00:02.412879 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.412860 6400 scope.go:122] "RemoveContainer" containerID="7e83f3b2e46a5a4f5cd47b7df1c455c5f522f6fa11427d9cdc4a0b0759f3d301" Nov 03 01:00:02.414568 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.414544 6400 generic.go:356] "Generic (PLEG): container finished" podID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" containerID="51df23c04b7b5e85c8c5283516363dbd40d479a17a81bce620f2746cc5411593" exitCode=60 Nov 03 01:00:02.414648 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.414629 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" event={"ID":"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2","Type":"ContainerDied","Data":"51df23c04b7b5e85c8c5283516363dbd40d479a17a81bce620f2746cc5411593"} Nov 03 01:00:02.414799 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:02.414780 6400 scope.go:122] "RemoveContainer" containerID="51df23c04b7b5e85c8c5283516363dbd40d479a17a81bce620f2746cc5411593" Nov 03 01:00:03.422228 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:03.422194 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" event={"ID":"0f55000e-0fcb-4045-a07b-7f89fe9f20c3","Type":"ContainerStarted","Data":"6418e377ccfedebe789051d17a8326ad61cdb935c7ddd90564814ecf09712734"} Nov 03 01:00:03.424057 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:03.424038 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" event={"ID":"f017544e-4cbe-4f21-abdc-70b6c6ab8981","Type":"ContainerStarted","Data":"c4e137b06c15d72bda2880dbd2d770d9bc19c5f9ccee7325d9083db1bc932471"} Nov 03 01:00:03.425878 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:03.425850 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" event={"ID":"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2","Type":"ContainerStarted","Data":"c742b3b0cf41108b081085cf97112125e61124e5a20fea4d9f3af939ac187b1b"} Nov 03 01:00:03.427570 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:03.427550 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" event={"ID":"780105c6-b952-4193-b287-9e93040ea9e6","Type":"ContainerStarted","Data":"472655349fa284618746a3e9866f68c919345f682708db0b874ec62ca7cb2a7f"} Nov 03 01:00:03.429268 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:03.429242 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" event={"ID":"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c","Type":"ContainerStarted","Data":"91474e6d9f8a8e02ad2bec523762d72d3358eaad861ac2d8fb8cc0c8eb74b42a"} Nov 03 01:00:03.431014 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:03.430990 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" event={"ID":"6da8cdf3-0d59-48ca-ad28-6a85adac1544","Type":"ContainerStarted","Data":"0b94c2e1e1a5ab4f54ea25c41de6ada0fb7e7de75efaca7debc34fd2936fbb84"} Nov 03 01:00:03.536812 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:03.536766 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" podStartSLOduration=3.536752733 podStartE2EDuration="3.536752733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.534569205 +0000 UTC m=+3250.485509363" watchObservedRunningTime="2026-11-03 01:00:03.536752733 +0000 UTC m=+3250.487692889" Nov 03 01:00:03.648457 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:03.648404 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" podStartSLOduration=3.648390589 podStartE2EDuration="3.648390589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.646484693 +0000 UTC m=+3250.597424851" watchObservedRunningTime="2026-11-03 01:00:03.648390589 +0000 UTC m=+3250.599330744" Nov 03 01:00:03.759475 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:03.759428 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" podStartSLOduration=3.7594103260000002 podStartE2EDuration="3.759410326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.758239118 +0000 UTC m=+3250.709179290" watchObservedRunningTime="2026-11-03 01:00:03.759410326 +0000 UTC m=+3250.710350490" Nov 03 01:00:03.873952 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:03.873895 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" podStartSLOduration=3.87380235 podStartE2EDuration="3.87380235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.871646324 +0000 UTC m=+3250.822586480" watchObservedRunningTime="2026-11-03 01:00:03.87380235 +0000 UTC m=+3250.824742506" Nov 03 01:00:03.987477 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:03.987380 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" podStartSLOduration=3.987367569 podStartE2EDuration="3.987367569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.98489462 +0000 UTC m=+3250.935834779" watchObservedRunningTime="2026-11-03 01:00:03.987367569 +0000 UTC m=+3250.938307725" Nov 03 01:00:04.098510 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.098462 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" podStartSLOduration=4.098446803 podStartE2EDuration="4.098446803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.096081472 +0000 UTC m=+3251.047021630" watchObservedRunningTime="2026-11-03 01:00:04.098446803 +0000 UTC m=+3251.049387048" Nov 03 01:00:04.437743 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.437706 6400 generic.go:356] "Generic (PLEG): container finished" podID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" containerID="6418e377ccfedebe789051d17a8326ad61cdb935c7ddd90564814ecf09712734" exitCode=60 Nov 03 01:00:04.438263 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.437776 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" event={"ID":"0f55000e-0fcb-4045-a07b-7f89fe9f20c3","Type":"ContainerDied","Data":"6418e377ccfedebe789051d17a8326ad61cdb935c7ddd90564814ecf09712734"} Nov 03 01:00:04.438263 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.437871 6400 scope.go:122] "RemoveContainer" containerID="a07eae6946e7d75effb755339de55f7ec24a508548b71becb9b5f6a82df6e3f6" Nov 03 01:00:04.438263 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.438197 6400 scope.go:122] "RemoveContainer" containerID="6418e377ccfedebe789051d17a8326ad61cdb935c7ddd90564814ecf09712734" Nov 03 01:00:04.438516 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:04.438493 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2_odh-ai-gateway-infra(0f55000e-0fcb-4045-a07b-7f89fe9f20c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" podUID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" Nov 03 01:00:04.439690 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.439665 6400 generic.go:356] "Generic (PLEG): container finished" podID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" containerID="c4e137b06c15d72bda2880dbd2d770d9bc19c5f9ccee7325d9083db1bc932471" exitCode=60 Nov 03 01:00:04.439789 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.439744 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" event={"ID":"f017544e-4cbe-4f21-abdc-70b6c6ab8981","Type":"ContainerDied","Data":"c4e137b06c15d72bda2880dbd2d770d9bc19c5f9ccee7325d9083db1bc932471"} Nov 03 01:00:04.440006 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.439989 6400 scope.go:122] "RemoveContainer" containerID="c4e137b06c15d72bda2880dbd2d770d9bc19c5f9ccee7325d9083db1bc932471" Nov 03 01:00:04.440276 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:04.440248 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2_odh-ai-gateway-infra(f017544e-4cbe-4f21-abdc-70b6c6ab8981)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" podUID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" Nov 03 01:00:04.441610 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.441590 6400 generic.go:356] "Generic (PLEG): container finished" podID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" containerID="c742b3b0cf41108b081085cf97112125e61124e5a20fea4d9f3af939ac187b1b" exitCode=60 Nov 03 01:00:04.441699 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.441673 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" event={"ID":"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2","Type":"ContainerDied","Data":"c742b3b0cf41108b081085cf97112125e61124e5a20fea4d9f3af939ac187b1b"} Nov 03 01:00:04.441883 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.441867 6400 scope.go:122] "RemoveContainer" containerID="c742b3b0cf41108b081085cf97112125e61124e5a20fea4d9f3af939ac187b1b" Nov 03 01:00:04.442124 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:04.442101 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w_odh-ai-gateway-infra(e4f2f5d3-8ed3-417f-b35f-81e082b77aa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" podUID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" Nov 03 01:00:04.443513 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.443469 6400 generic.go:356] "Generic (PLEG): container finished" podID="780105c6-b952-4193-b287-9e93040ea9e6" containerID="472655349fa284618746a3e9866f68c919345f682708db0b874ec62ca7cb2a7f" exitCode=60 Nov 03 01:00:04.443610 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.443590 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" event={"ID":"780105c6-b952-4193-b287-9e93040ea9e6","Type":"ContainerDied","Data":"472655349fa284618746a3e9866f68c919345f682708db0b874ec62ca7cb2a7f"} Nov 03 01:00:04.443837 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.443809 6400 scope.go:122] "RemoveContainer" containerID="472655349fa284618746a3e9866f68c919345f682708db0b874ec62ca7cb2a7f" Nov 03 01:00:04.444067 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:04.444046 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x_odh-ai-gateway-infra(780105c6-b952-4193-b287-9e93040ea9e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" podUID="780105c6-b952-4193-b287-9e93040ea9e6" Nov 03 01:00:04.445506 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.445485 6400 generic.go:356] "Generic (PLEG): container finished" podID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" containerID="91474e6d9f8a8e02ad2bec523762d72d3358eaad861ac2d8fb8cc0c8eb74b42a" exitCode=60 Nov 03 01:00:04.445583 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.445561 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" event={"ID":"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c","Type":"ContainerDied","Data":"91474e6d9f8a8e02ad2bec523762d72d3358eaad861ac2d8fb8cc0c8eb74b42a"} Nov 03 01:00:04.445733 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.445719 6400 scope.go:122] "RemoveContainer" containerID="91474e6d9f8a8e02ad2bec523762d72d3358eaad861ac2d8fb8cc0c8eb74b42a" Nov 03 01:00:04.445959 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:04.445936 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm_odh-ai-gateway-infra(1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" podUID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" Nov 03 01:00:04.447308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.447285 6400 generic.go:356] "Generic (PLEG): container finished" podID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" containerID="0b94c2e1e1a5ab4f54ea25c41de6ada0fb7e7de75efaca7debc34fd2936fbb84" exitCode=60 Nov 03 01:00:04.447388 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.447329 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" event={"ID":"6da8cdf3-0d59-48ca-ad28-6a85adac1544","Type":"ContainerDied","Data":"0b94c2e1e1a5ab4f54ea25c41de6ada0fb7e7de75efaca7debc34fd2936fbb84"} Nov 03 01:00:04.447510 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.447498 6400 scope.go:122] "RemoveContainer" containerID="0b94c2e1e1a5ab4f54ea25c41de6ada0fb7e7de75efaca7debc34fd2936fbb84" Nov 03 01:00:04.447679 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:04.447663 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4_odh-ai-gateway-infra(6da8cdf3-0d59-48ca-ad28-6a85adac1544)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" podUID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" Nov 03 01:00:04.453393 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.453374 6400 scope.go:122] "RemoveContainer" containerID="7e83f3b2e46a5a4f5cd47b7df1c455c5f522f6fa11427d9cdc4a0b0759f3d301" Nov 03 01:00:04.470452 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.470425 6400 scope.go:122] "RemoveContainer" containerID="51df23c04b7b5e85c8c5283516363dbd40d479a17a81bce620f2746cc5411593" Nov 03 01:00:04.483703 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.483672 6400 scope.go:122] "RemoveContainer" containerID="22b04e53d468f3f2aa00709b92da5d089717830f126053c043347f26957c63c9" Nov 03 01:00:04.496426 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.496409 6400 scope.go:122] "RemoveContainer" containerID="4699dbd6c28d2c2ae4332cb4c6cbdea2816d97357f7cb77f4860864af1ae8684" Nov 03 01:00:04.508906 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:04.508889 6400 scope.go:122] "RemoveContainer" containerID="d445c9f7e18f7acf4a7b289966c1c45dbe246fdcff320f6a75aadd72bfbe7d7a" Nov 03 01:00:05.455147 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:05.455120 6400 scope.go:122] "RemoveContainer" containerID="6418e377ccfedebe789051d17a8326ad61cdb935c7ddd90564814ecf09712734" Nov 03 01:00:05.455591 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:05.455459 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2_odh-ai-gateway-infra(0f55000e-0fcb-4045-a07b-7f89fe9f20c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" podUID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" Nov 03 01:00:05.456639 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:05.456620 6400 scope.go:122] "RemoveContainer" containerID="c4e137b06c15d72bda2880dbd2d770d9bc19c5f9ccee7325d9083db1bc932471" Nov 03 01:00:05.456925 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:05.456906 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2_odh-ai-gateway-infra(f017544e-4cbe-4f21-abdc-70b6c6ab8981)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" podUID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" Nov 03 01:00:05.458177 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:05.458155 6400 scope.go:122] "RemoveContainer" containerID="c742b3b0cf41108b081085cf97112125e61124e5a20fea4d9f3af939ac187b1b" Nov 03 01:00:05.458425 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:05.458402 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w_odh-ai-gateway-infra(e4f2f5d3-8ed3-417f-b35f-81e082b77aa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" podUID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" Nov 03 01:00:05.459585 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:05.459569 6400 scope.go:122] "RemoveContainer" containerID="472655349fa284618746a3e9866f68c919345f682708db0b874ec62ca7cb2a7f" Nov 03 01:00:05.459798 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:05.459781 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x_odh-ai-gateway-infra(780105c6-b952-4193-b287-9e93040ea9e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" podUID="780105c6-b952-4193-b287-9e93040ea9e6" Nov 03 01:00:05.460932 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:05.460914 6400 scope.go:122] "RemoveContainer" containerID="91474e6d9f8a8e02ad2bec523762d72d3358eaad861ac2d8fb8cc0c8eb74b42a" Nov 03 01:00:05.461147 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:05.461130 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm_odh-ai-gateway-infra(1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" podUID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" Nov 03 01:00:05.462211 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:05.462198 6400 scope.go:122] "RemoveContainer" containerID="0b94c2e1e1a5ab4f54ea25c41de6ada0fb7e7de75efaca7debc34fd2936fbb84" Nov 03 01:00:05.462361 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:05.462347 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4_odh-ai-gateway-infra(6da8cdf3-0d59-48ca-ad28-6a85adac1544)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" podUID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" Nov 03 01:00:05.801538 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:05.801504 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:08.580984 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:08.580951 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:00:15.172397 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:15.172368 6400 scope.go:122] "RemoveContainer" containerID="c742b3b0cf41108b081085cf97112125e61124e5a20fea4d9f3af939ac187b1b" Nov 03 01:00:15.519603 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:15.519580 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" event={"ID":"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2","Type":"ContainerStarted","Data":"eb011ca11f8900f32f0ab844f0700bcabcac6c237f924166be0150a72e33e005"} Nov 03 01:00:16.172686 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:16.172658 6400 scope.go:122] "RemoveContainer" containerID="0b94c2e1e1a5ab4f54ea25c41de6ada0fb7e7de75efaca7debc34fd2936fbb84" Nov 03 01:00:16.173108 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:16.172858 6400 scope.go:122] "RemoveContainer" containerID="6418e377ccfedebe789051d17a8326ad61cdb935c7ddd90564814ecf09712734" Nov 03 01:00:16.173108 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:16.172975 6400 scope.go:122] "RemoveContainer" containerID="91474e6d9f8a8e02ad2bec523762d72d3358eaad861ac2d8fb8cc0c8eb74b42a" Nov 03 01:00:16.173108 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:16.173079 6400 scope.go:122] "RemoveContainer" containerID="c4e137b06c15d72bda2880dbd2d770d9bc19c5f9ccee7325d9083db1bc932471" Nov 03 01:00:16.173698 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:16.173333 6400 scope.go:122] "RemoveContainer" containerID="472655349fa284618746a3e9866f68c919345f682708db0b874ec62ca7cb2a7f" Nov 03 01:00:16.526964 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:16.526940 6400 generic.go:356] "Generic (PLEG): container finished" podID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" containerID="eb011ca11f8900f32f0ab844f0700bcabcac6c237f924166be0150a72e33e005" exitCode=60 Nov 03 01:00:16.527076 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:16.526991 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" event={"ID":"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2","Type":"ContainerDied","Data":"eb011ca11f8900f32f0ab844f0700bcabcac6c237f924166be0150a72e33e005"} Nov 03 01:00:16.527076 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:16.527039 6400 scope.go:122] "RemoveContainer" containerID="c742b3b0cf41108b081085cf97112125e61124e5a20fea4d9f3af939ac187b1b" Nov 03 01:00:16.527360 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:16.527345 6400 scope.go:122] "RemoveContainer" containerID="eb011ca11f8900f32f0ab844f0700bcabcac6c237f924166be0150a72e33e005" Nov 03 01:00:16.527616 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:16.527599 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w_odh-ai-gateway-infra(e4f2f5d3-8ed3-417f-b35f-81e082b77aa2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" podUID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" Nov 03 01:00:16.649221 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:16.649158 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w"] Nov 03 01:00:16.990999 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:16.990925 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:17.534078 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.534053 6400 generic.go:356] "Generic (PLEG): container finished" podID="780105c6-b952-4193-b287-9e93040ea9e6" containerID="0e7759a652049c0812957f806813e1ffce54a7bfbe4e644d86184bbb9c099916" exitCode=60 Nov 03 01:00:17.534486 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.534127 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" event={"ID":"780105c6-b952-4193-b287-9e93040ea9e6","Type":"ContainerDied","Data":"0e7759a652049c0812957f806813e1ffce54a7bfbe4e644d86184bbb9c099916"} Nov 03 01:00:17.534486 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.534185 6400 scope.go:122] "RemoveContainer" containerID="472655349fa284618746a3e9866f68c919345f682708db0b874ec62ca7cb2a7f" Nov 03 01:00:17.534603 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.534534 6400 scope.go:122] "RemoveContainer" containerID="0e7759a652049c0812957f806813e1ffce54a7bfbe4e644d86184bbb9c099916" Nov 03 01:00:17.534835 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:17.534796 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x_odh-ai-gateway-infra(780105c6-b952-4193-b287-9e93040ea9e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" podUID="780105c6-b952-4193-b287-9e93040ea9e6" Nov 03 01:00:17.536174 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.536157 6400 generic.go:356] "Generic (PLEG): container finished" podID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" containerID="bd78308d903cfdaf78d6891164f0f37653e24f488aa8b1103dd361dc1c173f8b" exitCode=60 Nov 03 01:00:17.536293 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.536222 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" event={"ID":"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c","Type":"ContainerDied","Data":"bd78308d903cfdaf78d6891164f0f37653e24f488aa8b1103dd361dc1c173f8b"} Nov 03 01:00:17.536546 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.536529 6400 scope.go:122] "RemoveContainer" containerID="bd78308d903cfdaf78d6891164f0f37653e24f488aa8b1103dd361dc1c173f8b" Nov 03 01:00:17.536800 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:17.536771 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm_odh-ai-gateway-infra(1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" podUID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" Nov 03 01:00:17.538309 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.537990 6400 generic.go:356] "Generic (PLEG): container finished" podID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" containerID="d23fc3b3ca70ecba2c29e173f7c5d30fd1ae7b392d25211d7c7fb9443160fa8d" exitCode=60 Nov 03 01:00:17.538309 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.538017 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" event={"ID":"6da8cdf3-0d59-48ca-ad28-6a85adac1544","Type":"ContainerDied","Data":"d23fc3b3ca70ecba2c29e173f7c5d30fd1ae7b392d25211d7c7fb9443160fa8d"} Nov 03 01:00:17.538452 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.538318 6400 scope.go:122] "RemoveContainer" containerID="d23fc3b3ca70ecba2c29e173f7c5d30fd1ae7b392d25211d7c7fb9443160fa8d" Nov 03 01:00:17.538550 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:17.538526 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4_odh-ai-gateway-infra(6da8cdf3-0d59-48ca-ad28-6a85adac1544)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" podUID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" Nov 03 01:00:17.540193 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.540174 6400 generic.go:356] "Generic (PLEG): container finished" podID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" containerID="0ca2f6dab5441f5df9973c9ff07e626bc88143efd35ccbba8fa8524c230c6c07" exitCode=60 Nov 03 01:00:17.540298 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.540234 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" event={"ID":"0f55000e-0fcb-4045-a07b-7f89fe9f20c3","Type":"ContainerDied","Data":"0ca2f6dab5441f5df9973c9ff07e626bc88143efd35ccbba8fa8524c230c6c07"} Nov 03 01:00:17.540504 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.540489 6400 scope.go:122] "RemoveContainer" containerID="0ca2f6dab5441f5df9973c9ff07e626bc88143efd35ccbba8fa8524c230c6c07" Nov 03 01:00:17.540694 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:17.540677 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2_odh-ai-gateway-infra(0f55000e-0fcb-4045-a07b-7f89fe9f20c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" podUID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" Nov 03 01:00:17.542176 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.542156 6400 generic.go:356] "Generic (PLEG): container finished" podID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" containerID="f4b9156ac003f7d0c054249d2eb0e3496e4d8017f202fb07cadd617b1ad709b6" exitCode=60 Nov 03 01:00:17.542285 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.542210 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" event={"ID":"f017544e-4cbe-4f21-abdc-70b6c6ab8981","Type":"ContainerDied","Data":"f4b9156ac003f7d0c054249d2eb0e3496e4d8017f202fb07cadd617b1ad709b6"} Nov 03 01:00:17.542534 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.542513 6400 scope.go:122] "RemoveContainer" containerID="f4b9156ac003f7d0c054249d2eb0e3496e4d8017f202fb07cadd617b1ad709b6" Nov 03 01:00:17.542745 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:17.542725 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2_odh-ai-gateway-infra(f017544e-4cbe-4f21-abdc-70b6c6ab8981)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" podUID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" Nov 03 01:00:17.547553 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.547535 6400 scope.go:122] "RemoveContainer" containerID="91474e6d9f8a8e02ad2bec523762d72d3358eaad861ac2d8fb8cc0c8eb74b42a" Nov 03 01:00:17.559678 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.559658 6400 scope.go:122] "RemoveContainer" containerID="0b94c2e1e1a5ab4f54ea25c41de6ada0fb7e7de75efaca7debc34fd2936fbb84" Nov 03 01:00:17.574808 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.574791 6400 scope.go:122] "RemoveContainer" containerID="6418e377ccfedebe789051d17a8326ad61cdb935c7ddd90564814ecf09712734" Nov 03 01:00:17.587913 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.587895 6400 scope.go:122] "RemoveContainer" containerID="c4e137b06c15d72bda2880dbd2d770d9bc19c5f9ccee7325d9083db1bc932471" Nov 03 01:00:17.685313 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.685292 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" Nov 03 01:00:17.853467 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.853408 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-maas-api-service-ca\") pod \"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2\" (UID: \"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2\") " Nov 03 01:00:17.853656 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.853495 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-kube-api-access-7xnx9\" (UniqueName: \"kubernetes.io/projected/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-kube-api-access-7xnx9\") pod \"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2\" (UID: \"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2\") " Nov 03 01:00:17.853935 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.853913 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-maas-api-service-ca" pod "e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" (UID: "e4f2f5d3-8ed3-417f-b35f-81e082b77aa2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:17.855700 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.855680 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-kube-api-access-7xnx9" pod "e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" (UID: "e4f2f5d3-8ed3-417f-b35f-81e082b77aa2"). InnerVolumeSpecName "kube-api-access-7xnx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:17.954458 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.954429 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:17.954458 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:17.954454 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xnx9\" (UniqueName: \"kubernetes.io/projected/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2-kube-api-access-7xnx9\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.557743 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.557719 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" Nov 03 01:00:18.558216 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.557781 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w" event={"ID":"e4f2f5d3-8ed3-417f-b35f-81e082b77aa2","Type":"ContainerDied","Data":"240673cad6d8e4e1ee39b60965fdd547c29372991c0bc387e1ea352987924dee"} Nov 03 01:00:18.558216 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.557861 6400 scope.go:122] "RemoveContainer" containerID="eb011ca11f8900f32f0ab844f0700bcabcac6c237f924166be0150a72e33e005" Nov 03 01:00:18.656213 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.656182 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x"] Nov 03 01:00:18.723707 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.723678 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w"] Nov 03 01:00:18.727115 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.727088 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ea50b7-29894460-8ns5w"] Nov 03 01:00:18.771248 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.771215 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2"] Nov 03 01:00:18.809971 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.809909 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" Nov 03 01:00:18.927847 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.927797 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" Nov 03 01:00:18.964576 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.964546 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780105c6-b952-4193-b287-9e93040ea9e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780105c6-b952-4193-b287-9e93040ea9e6-maas-api-service-ca\") pod \"780105c6-b952-4193-b287-9e93040ea9e6\" (UID: \"780105c6-b952-4193-b287-9e93040ea9e6\") " Nov 03 01:00:18.964756 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.964640 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780105c6-b952-4193-b287-9e93040ea9e6-kube-api-access-k9lz7\" (UniqueName: \"kubernetes.io/projected/780105c6-b952-4193-b287-9e93040ea9e6-kube-api-access-k9lz7\") pod \"780105c6-b952-4193-b287-9e93040ea9e6\" (UID: \"780105c6-b952-4193-b287-9e93040ea9e6\") " Nov 03 01:00:18.964925 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.964904 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780105c6-b952-4193-b287-9e93040ea9e6-maas-api-service-ca" pod "780105c6-b952-4193-b287-9e93040ea9e6" (UID: "780105c6-b952-4193-b287-9e93040ea9e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:18.966930 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.966907 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780105c6-b952-4193-b287-9e93040ea9e6-kube-api-access-k9lz7" pod "780105c6-b952-4193-b287-9e93040ea9e6" (UID: "780105c6-b952-4193-b287-9e93040ea9e6"). InnerVolumeSpecName "kube-api-access-k9lz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:18.996689 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:18.996662 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4"] Nov 03 01:00:19.065422 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.065351 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-kube-api-access-krnsn\" (UniqueName: \"kubernetes.io/projected/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-kube-api-access-krnsn\") pod \"0f55000e-0fcb-4045-a07b-7f89fe9f20c3\" (UID: \"0f55000e-0fcb-4045-a07b-7f89fe9f20c3\") " Nov 03 01:00:19.065422 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.065407 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-maas-api-service-ca\") pod \"0f55000e-0fcb-4045-a07b-7f89fe9f20c3\" (UID: \"0f55000e-0fcb-4045-a07b-7f89fe9f20c3\") " Nov 03 01:00:19.065961 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.065936 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780105c6-b952-4193-b287-9e93040ea9e6-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:19.066079 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.065963 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9lz7\" (UniqueName: \"kubernetes.io/projected/780105c6-b952-4193-b287-9e93040ea9e6-kube-api-access-k9lz7\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:19.066310 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.066284 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-maas-api-service-ca" pod "0f55000e-0fcb-4045-a07b-7f89fe9f20c3" (UID: "0f55000e-0fcb-4045-a07b-7f89fe9f20c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:19.072084 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.072033 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-kube-api-access-krnsn" pod "0f55000e-0fcb-4045-a07b-7f89fe9f20c3" (UID: "0f55000e-0fcb-4045-a07b-7f89fe9f20c3"). InnerVolumeSpecName "kube-api-access-krnsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:19.107484 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.107454 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2"] Nov 03 01:00:19.127236 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.127211 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" Nov 03 01:00:19.168211 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.167956 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krnsn\" (UniqueName: \"kubernetes.io/projected/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-kube-api-access-krnsn\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:19.168211 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.167989 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f55000e-0fcb-4045-a07b-7f89fe9f20c3-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:19.188855 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.187933 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" path="/var/lib/kubelet/pods/e4f2f5d3-8ed3-417f-b35f-81e082b77aa2/volumes" Nov 03 01:00:19.219433 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.219399 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm"] Nov 03 01:00:19.239908 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.239884 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" Nov 03 01:00:19.269019 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.268991 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da8cdf3-0d59-48ca-ad28-6a85adac1544-kube-api-access-tknjf\" (UniqueName: \"kubernetes.io/projected/6da8cdf3-0d59-48ca-ad28-6a85adac1544-kube-api-access-tknjf\") pod \"6da8cdf3-0d59-48ca-ad28-6a85adac1544\" (UID: \"6da8cdf3-0d59-48ca-ad28-6a85adac1544\") " Nov 03 01:00:19.269214 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.269155 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da8cdf3-0d59-48ca-ad28-6a85adac1544-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da8cdf3-0d59-48ca-ad28-6a85adac1544-maas-api-service-ca\") pod \"6da8cdf3-0d59-48ca-ad28-6a85adac1544\" (UID: \"6da8cdf3-0d59-48ca-ad28-6a85adac1544\") " Nov 03 01:00:19.269570 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.269544 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da8cdf3-0d59-48ca-ad28-6a85adac1544-maas-api-service-ca" pod "6da8cdf3-0d59-48ca-ad28-6a85adac1544" (UID: "6da8cdf3-0d59-48ca-ad28-6a85adac1544"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:19.271212 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.271184 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da8cdf3-0d59-48ca-ad28-6a85adac1544-kube-api-access-tknjf" pod "6da8cdf3-0d59-48ca-ad28-6a85adac1544" (UID: "6da8cdf3-0d59-48ca-ad28-6a85adac1544"). InnerVolumeSpecName "kube-api-access-tknjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:19.363511 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.363484 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" Nov 03 01:00:19.370091 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.370066 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f017544e-4cbe-4f21-abdc-70b6c6ab8981-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f017544e-4cbe-4f21-abdc-70b6c6ab8981-maas-api-service-ca\") pod \"f017544e-4cbe-4f21-abdc-70b6c6ab8981\" (UID: \"f017544e-4cbe-4f21-abdc-70b6c6ab8981\") " Nov 03 01:00:19.370243 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.370168 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f017544e-4cbe-4f21-abdc-70b6c6ab8981-kube-api-access-wbq9x\" (UniqueName: \"kubernetes.io/projected/f017544e-4cbe-4f21-abdc-70b6c6ab8981-kube-api-access-wbq9x\") pod \"f017544e-4cbe-4f21-abdc-70b6c6ab8981\" (UID: \"f017544e-4cbe-4f21-abdc-70b6c6ab8981\") " Nov 03 01:00:19.370449 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.370415 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f017544e-4cbe-4f21-abdc-70b6c6ab8981-maas-api-service-ca" pod "f017544e-4cbe-4f21-abdc-70b6c6ab8981" (UID: "f017544e-4cbe-4f21-abdc-70b6c6ab8981"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:19.370553 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.370524 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tknjf\" (UniqueName: \"kubernetes.io/projected/6da8cdf3-0d59-48ca-ad28-6a85adac1544-kube-api-access-tknjf\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:19.370553 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.370538 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f017544e-4cbe-4f21-abdc-70b6c6ab8981-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:19.370553 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.370547 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da8cdf3-0d59-48ca-ad28-6a85adac1544-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:19.372531 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.372504 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f017544e-4cbe-4f21-abdc-70b6c6ab8981-kube-api-access-wbq9x" pod "f017544e-4cbe-4f21-abdc-70b6c6ab8981" (UID: "f017544e-4cbe-4f21-abdc-70b6c6ab8981"). InnerVolumeSpecName "kube-api-access-wbq9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:19.471563 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.471534 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-maas-api-service-ca\") pod \"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c\" (UID: \"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c\") " Nov 03 01:00:19.471741 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.471674 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-kube-api-access-trtf7\" (UniqueName: \"kubernetes.io/projected/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-kube-api-access-trtf7\") pod \"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c\" (UID: \"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c\") " Nov 03 01:00:19.471918 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.471896 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-maas-api-service-ca" pod "1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" (UID: "1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:19.472101 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.472089 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbq9x\" (UniqueName: \"kubernetes.io/projected/f017544e-4cbe-4f21-abdc-70b6c6ab8981-kube-api-access-wbq9x\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:19.472156 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.472104 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:19.473845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.473808 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-kube-api-access-trtf7" pod "1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" (UID: "1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c"). InnerVolumeSpecName "kube-api-access-trtf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:19.567262 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.567232 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" Nov 03 01:00:19.567726 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.567259 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x" event={"ID":"780105c6-b952-4193-b287-9e93040ea9e6","Type":"ContainerDied","Data":"a6d3598c1a524f7782a580f0b8a227c61abf52276f09f8d3bb6bb927b7e479f6"} Nov 03 01:00:19.567726 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.567323 6400 scope.go:122] "RemoveContainer" containerID="0e7759a652049c0812957f806813e1ffce54a7bfbe4e644d86184bbb9c099916" Nov 03 01:00:19.568802 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.568784 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" Nov 03 01:00:19.568802 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.568790 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm" event={"ID":"1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c","Type":"ContainerDied","Data":"3886560a5266ebc5446dbd1b639b0c249fd71cb93bdce8226693a4d1610660b5"} Nov 03 01:00:19.571102 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.570870 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" event={"ID":"6da8cdf3-0d59-48ca-ad28-6a85adac1544","Type":"ContainerDied","Data":"172429d96373e23cd5a53385ae8d61b7f1c524cb37803a590e4dbba77d22baec"} Nov 03 01:00:19.571102 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.570873 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4" Nov 03 01:00:19.572401 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.572367 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" event={"ID":"0f55000e-0fcb-4045-a07b-7f89fe9f20c3","Type":"ContainerDied","Data":"00b39f1c5b59c381e1f7f1a11bd8afa580221635cc9f7dc42a71e5338db9068e"} Nov 03 01:00:19.572513 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.572375 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2" Nov 03 01:00:19.573054 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.573035 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trtf7\" (UniqueName: \"kubernetes.io/projected/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c-kube-api-access-trtf7\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:19.577196 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.577179 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" Nov 03 01:00:19.577302 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.577197 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2" event={"ID":"f017544e-4cbe-4f21-abdc-70b6c6ab8981","Type":"ContainerDied","Data":"84367fda87f8c95e7a2e60af70297c1d2e20ae3bfb3732dd6fead39812722468"} Nov 03 01:00:19.578668 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.578641 6400 scope.go:122] "RemoveContainer" containerID="bd78308d903cfdaf78d6891164f0f37653e24f488aa8b1103dd361dc1c173f8b" Nov 03 01:00:19.589053 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.589031 6400 scope.go:122] "RemoveContainer" containerID="d23fc3b3ca70ecba2c29e173f7c5d30fd1ae7b392d25211d7c7fb9443160fa8d" Nov 03 01:00:19.597419 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.597400 6400 scope.go:122] "RemoveContainer" containerID="0ca2f6dab5441f5df9973c9ff07e626bc88143efd35ccbba8fa8524c230c6c07" Nov 03 01:00:19.605685 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.605670 6400 scope.go:122] "RemoveContainer" containerID="f4b9156ac003f7d0c054249d2eb0e3496e4d8017f202fb07cadd617b1ad709b6" Nov 03 01:00:19.743262 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.743230 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2"] Nov 03 01:00:19.747050 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.747025 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f40c6c-29894460-mswc2"] Nov 03 01:00:19.854870 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.854845 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4"] Nov 03 01:00:19.858411 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.858356 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-115ee5-29894460-cvmc4"] Nov 03 01:00:19.966082 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.966055 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2"] Nov 03 01:00:19.976095 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:19.976073 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894460-nz5c2"] Nov 03 01:00:20.083184 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:20.083155 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x"] Nov 03 01:00:20.086845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:20.086795 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894460-lct6x"] Nov 03 01:00:20.254235 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:20.254149 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm"] Nov 03 01:00:20.261631 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:20.261603 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-087f6c-29894460-j2wzm"] Nov 03 01:00:20.648192 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:20.648160 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:21.179307 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:21.179270 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" path="/var/lib/kubelet/pods/0f55000e-0fcb-4045-a07b-7f89fe9f20c3/volumes" Nov 03 01:00:21.179717 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:21.179703 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" path="/var/lib/kubelet/pods/1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c/volumes" Nov 03 01:00:21.180087 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:21.180073 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" path="/var/lib/kubelet/pods/6da8cdf3-0d59-48ca-ad28-6a85adac1544/volumes" Nov 03 01:00:21.180710 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:21.180689 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780105c6-b952-4193-b287-9e93040ea9e6" path="/var/lib/kubelet/pods/780105c6-b952-4193-b287-9e93040ea9e6/volumes" Nov 03 01:00:21.181066 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:21.181053 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" path="/var/lib/kubelet/pods/f017544e-4cbe-4f21-abdc-70b6c6ab8981/volumes" Nov 03 01:00:25.997562 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.997524 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj"] Nov 03 01:00:25.998077 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998060 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" containerName="cleanup" Nov 03 01:00:25.998122 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998078 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" containerName="cleanup" Nov 03 01:00:25.998122 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998094 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" containerName="cleanup" Nov 03 01:00:25.998122 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998099 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" containerName="cleanup" Nov 03 01:00:25.998122 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998107 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" containerName="cleanup" Nov 03 01:00:25.998122 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998112 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" containerName="cleanup" Nov 03 01:00:25.998122 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998124 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998128 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998134 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998138 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998148 6400 cpu_manager.go:414] "containerMap: removing container" podUID="780105c6-b952-4193-b287-9e93040ea9e6" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998152 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780105c6-b952-4193-b287-9e93040ea9e6" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998160 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998164 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998172 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998176 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998183 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998187 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998198 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998203 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998209 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998213 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998219 6400 cpu_manager.go:414] "containerMap: removing container" podUID="780105c6-b952-4193-b287-9e93040ea9e6" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998223 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780105c6-b952-4193-b287-9e93040ea9e6" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998229 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998233 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998239 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998243 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" containerName="cleanup" Nov 03 01:00:25.998308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998310 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" containerName="cleanup" Nov 03 01:00:25.998974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998320 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" containerName="cleanup" Nov 03 01:00:25.998974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998326 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" containerName="cleanup" Nov 03 01:00:25.998974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998333 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" containerName="cleanup" Nov 03 01:00:25.998974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998338 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" containerName="cleanup" Nov 03 01:00:25.998974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998343 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="780105c6-b952-4193-b287-9e93040ea9e6" containerName="cleanup" Nov 03 01:00:25.998974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998347 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" containerName="cleanup" Nov 03 01:00:25.998974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998352 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" containerName="cleanup" Nov 03 01:00:25.998974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998356 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" containerName="cleanup" Nov 03 01:00:25.998974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998361 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" containerName="cleanup" Nov 03 01:00:25.998974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998367 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" containerName="cleanup" Nov 03 01:00:25.998974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998372 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="780105c6-b952-4193-b287-9e93040ea9e6" containerName="cleanup" Nov 03 01:00:25.998974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998376 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f647fa6-cfa8-4cb7-b8dd-046eb07dcf5c" containerName="cleanup" Nov 03 01:00:25.998974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:25.998381 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" containerName="cleanup" Nov 03 01:00:26.001586 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.001570 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.045097 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.045036 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhg5k\" (UniqueName: \"kubernetes.io/projected/8e8005c1-0893-45e6-82d5-3148021e8369-kube-api-access-nhg5k\") pod \"maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.045334 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.045138 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-maas-api-tls\") pod \"maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.045334 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.045189 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-metrics-certs\") pod \"maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.045334 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.045224 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8005c1-0893-45e6-82d5-3148021e8369-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.054882 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.054769 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-642e6a-serving-cert\"" Nov 03 01:00:26.057726 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.057689 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj"] Nov 03 01:00:26.146468 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.146432 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhg5k\" (UniqueName: \"kubernetes.io/projected/8e8005c1-0893-45e6-82d5-3148021e8369-kube-api-access-nhg5k\") pod \"maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.146468 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.146470 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-maas-api-tls\") pod \"maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.146735 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.146512 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-metrics-certs\") pod \"maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.146735 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.146541 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8005c1-0893-45e6-82d5-3148021e8369-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.146735 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:26.146643 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-serving-cert: secret "maas-api-e2e-extmod-b-642e6a-serving-cert" not found Nov 03 01:00:26.146735 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:26.146721 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-maas-api-tls podName:8e8005c1-0893-45e6-82d5-3148021e8369 nodeName:}" failed. No retries permitted until 2026-11-03 01:00:26.646702341 +0000 UTC m=+3273.597642476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-maas-api-tls") pod "maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" (UID: "8e8005c1-0893-45e6-82d5-3148021e8369") : secret "maas-api-e2e-extmod-b-642e6a-serving-cert" not found Nov 03 01:00:26.147316 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.147295 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8005c1-0893-45e6-82d5-3148021e8369-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.149143 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.149120 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-metrics-certs\") pod \"maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.205836 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.205792 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhg5k\" (UniqueName: \"kubernetes.io/projected/8e8005c1-0893-45e6-82d5-3148021e8369-kube-api-access-nhg5k\") pod \"maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.265608 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.265525 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:26.434318 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.434291 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:26.632404 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.632359 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:26.662508 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.662477 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-maas-api-tls\") pod \"maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.665145 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.665117 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-maas-api-tls\") pod \"maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:26.915061 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:26.914974 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:27.155694 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:27.155660 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:00:27.285683 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:27.285663 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj"] Nov 03 01:00:27.288100 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:00:27.288075 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8005c1_0893_45e6_82d5_3148021e8369.slice/crio-73077f8967731e56538d233800b1598fdfb1643514e072a8895999408c30236b WatchSource:0}: Error finding container 73077f8967731e56538d233800b1598fdfb1643514e072a8895999408c30236b: Status 404 returned error can't find the container with id 73077f8967731e56538d233800b1598fdfb1643514e072a8895999408c30236b Nov 03 01:00:27.634758 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:27.634681 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" event={"ID":"8e8005c1-0893-45e6-82d5-3148021e8369","Type":"ContainerStarted","Data":"73077f8967731e56538d233800b1598fdfb1643514e072a8895999408c30236b"} Nov 03 01:00:28.641952 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:28.641918 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" event={"ID":"8e8005c1-0893-45e6-82d5-3148021e8369","Type":"ContainerStarted","Data":"73383b44c40d339bd53f2f757569f9b2063de17d9eb6e7b6d6703da88e505939"} Nov 03 01:00:28.642354 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:28.642031 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:28.759499 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:28.759440 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" podStartSLOduration=2.8044006809999997 podStartE2EDuration="3.759420374s" totalImagesPullingTime="955.019693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:28.759046347 +0000 UTC m=+3275.709986504" watchObservedRunningTime="2026-11-03 01:00:28.759420374 +0000 UTC m=+3275.710360530" Nov 03 01:00:31.255786 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:31.255753 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:34.653635 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:34.653608 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:00:36.222010 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:36.221973 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:40.912809 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:40.912765 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:45.706558 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:45.706522 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:45.911615 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:45.911583 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:49.481130 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:49.481084 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:00:50.108600 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:50.108557 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:50.426567 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:50.426485 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:51.427899 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:51.427861 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:52.509182 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:52.509147 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:52.668748 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:52.668707 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:55.564095 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:55.564058 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:56.143328 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:56.143292 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:56.320129 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:56.320079 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:57.216471 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:00:57.216067 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:00:57.333100 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:00:57.333063 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:00.235218 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:00.235124 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:00.861471 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:00.861439 6400 generic.go:356] "Generic (PLEG): container finished" podID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerID="df6735d6ecd70913368ee05b400ad16ec3eed9791ea809484bce14479d17b092" exitCode=0 Nov 03 01:01:00.861669 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:00.861525 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" event={"ID":"bdae0d76-2372-4d89-89e2-84e69f787b99","Type":"ContainerDied","Data":"df6735d6ecd70913368ee05b400ad16ec3eed9791ea809484bce14479d17b092"} Nov 03 01:01:00.862028 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:00.862014 6400 scope.go:122] "RemoveContainer" containerID="df6735d6ecd70913368ee05b400ad16ec3eed9791ea809484bce14479d17b092" Nov 03 01:01:00.863838 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:00.863792 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:01:00.863916 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:00.863899 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:01:00.863953 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:00.863933 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:01:01.829815 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:01.829776 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:01.869371 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:01.869336 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" event={"ID":"bdae0d76-2372-4d89-89e2-84e69f787b99","Type":"ContainerStarted","Data":"b6314d9664b8eb08ac4d6b2c9503b3c23ccac4d61b528bdac8b03a9216a13099"} Nov 03 01:01:02.321853 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:02.321779 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 01:01:02.322988 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:02.322965 6400 patch_prober.go:29] interesting pod/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 01:01:02.323116 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:02.323037 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" podUID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 01:01:03.321628 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:03.321590 6400 patch_prober.go:29] interesting pod/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 01:01:03.322072 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:03.321656 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" podUID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 01:01:04.322300 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:04.322261 6400 patch_prober.go:29] interesting pod/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 01:01:04.322980 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:04.322949 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" podUID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 01:01:04.861570 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:04.861534 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 01:01:05.008088 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:05.008053 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:05.321735 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:05.321702 6400 patch_prober.go:29] interesting pod/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 01:01:05.321950 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:05.321754 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" podUID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 01:01:06.321726 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:06.321691 6400 patch_prober.go:29] interesting pod/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 01:01:06.322140 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:06.321753 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" podUID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 01:01:06.800963 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:06.800921 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:06.966150 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:06.966110 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:07.326095 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:07.326064 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 01:01:07.327235 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:07.327217 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 01:01:08.790298 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:08.790241 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:09.394489 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:09.394445 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:11.207161 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:11.207123 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:14.922389 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:14.922351 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:15.661871 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:15.661835 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:15.858501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:15.858462 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:16.232832 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:16.232797 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:16.653287 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:16.653253 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:16.828714 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:16.828672 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:17.387599 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:17.387560 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:18.211927 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:18.211892 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:18.838550 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:18.838498 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:20.559968 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:20.559939 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:20.729033 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:20.729001 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:21.333571 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:21.333535 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:21.590128 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:21.590037 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:21.822524 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:21.822491 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:26.036693 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:26.036637 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:26.274411 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:26.274373 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:26.481439 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:26.481354 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:27.154190 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:01:27.154145 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:01:27.909815 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:27.909782 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:28.682098 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:28.682056 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:29.432405 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:29.432372 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:29.674963 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:29.674926 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:32.972963 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:32.972869 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:33.771874 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:33.771838 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:01:35.319420 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:35.319388 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:35.476946 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:35.476908 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:36.252194 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:36.252157 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:36.885253 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:36.885222 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:37.400489 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:37.400444 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:37.574759 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:37.569429 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:38.230274 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:38.230240 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:41.215872 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:41.215833 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:41.404882 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:41.401617 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:42.089078 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:42.089043 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:42.816136 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:42.816098 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:43.342803 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:43.342772 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:43.497879 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:43.497838 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:44.351011 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:44.350973 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:44.690549 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:44.690449 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:47.664563 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:47.664523 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:49.384436 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:49.384394 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:53.290033 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:53.289983 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:53.444725 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:53.444673 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:54.195000 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:54.194970 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:54.924244 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:54.923876 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:55.367148 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:55.367105 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:56.224887 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:56.224851 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:57.152242 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:01:57.152204 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:01:59.402209 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:01:59.402178 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:01.987940 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:01.987898 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:03.369209 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:03.369178 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:03.626239 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:03.626158 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:04.371606 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:04.371563 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:07.593765 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:07.593729 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:09.229522 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:09.229483 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:13.249363 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.249323 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh"] Nov 03 01:02:13.249885 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.249526 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" podUID="87a3a116-a7eb-4e1e-aea8-113107841624" containerName="maas-api" containerID="cri-o://90064bb842370308cd7d6abb2679e7004d34351668345305eee7cb0085839a9f" gracePeriod=30 Nov 03 01:02:13.376857 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.376748 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:13.600749 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.600718 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 01:02:13.785028 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.784990 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-maas-api-tls\") pod \"87a3a116-a7eb-4e1e-aea8-113107841624\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " Nov 03 01:02:13.785256 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.785106 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-metrics-certs\") pod \"87a3a116-a7eb-4e1e-aea8-113107841624\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " Nov 03 01:02:13.785256 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.785183 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a3a116-a7eb-4e1e-aea8-113107841624-kube-api-access-d55gp\" (UniqueName: \"kubernetes.io/projected/87a3a116-a7eb-4e1e-aea8-113107841624-kube-api-access-d55gp\") pod \"87a3a116-a7eb-4e1e-aea8-113107841624\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " Nov 03 01:02:13.785256 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.785227 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a3a116-a7eb-4e1e-aea8-113107841624-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a3a116-a7eb-4e1e-aea8-113107841624-openshift-service-ca\") pod \"87a3a116-a7eb-4e1e-aea8-113107841624\" (UID: \"87a3a116-a7eb-4e1e-aea8-113107841624\") " Nov 03 01:02:13.786125 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.786096 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a3a116-a7eb-4e1e-aea8-113107841624-openshift-service-ca" pod "87a3a116-a7eb-4e1e-aea8-113107841624" (UID: "87a3a116-a7eb-4e1e-aea8-113107841624"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:13.788800 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.788756 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-maas-api-tls" pod "87a3a116-a7eb-4e1e-aea8-113107841624" (UID: "87a3a116-a7eb-4e1e-aea8-113107841624"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:13.789070 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.789035 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-metrics-certs" pod "87a3a116-a7eb-4e1e-aea8-113107841624" (UID: "87a3a116-a7eb-4e1e-aea8-113107841624"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:13.804790 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.804758 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a3a116-a7eb-4e1e-aea8-113107841624-kube-api-access-d55gp" pod "87a3a116-a7eb-4e1e-aea8-113107841624" (UID: "87a3a116-a7eb-4e1e-aea8-113107841624"). InnerVolumeSpecName "kube-api-access-d55gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:13.886734 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.886629 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:13.886734 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.886668 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d55gp\" (UniqueName: \"kubernetes.io/projected/87a3a116-a7eb-4e1e-aea8-113107841624-kube-api-access-d55gp\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:13.886734 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.886682 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a3a116-a7eb-4e1e-aea8-113107841624-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:13.886734 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:13.886694 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a3a116-a7eb-4e1e-aea8-113107841624-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:14.337793 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:14.337754 6400 generic.go:356] "Generic (PLEG): container finished" podID="87a3a116-a7eb-4e1e-aea8-113107841624" containerID="90064bb842370308cd7d6abb2679e7004d34351668345305eee7cb0085839a9f" exitCode=0 Nov 03 01:02:14.337793 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:14.337796 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" event={"ID":"87a3a116-a7eb-4e1e-aea8-113107841624","Type":"ContainerDied","Data":"90064bb842370308cd7d6abb2679e7004d34351668345305eee7cb0085839a9f"} Nov 03 01:02:14.338338 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:14.337839 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" Nov 03 01:02:14.338338 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:14.337860 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh" event={"ID":"87a3a116-a7eb-4e1e-aea8-113107841624","Type":"ContainerDied","Data":"44dc0984b1e26fe3bf1353700c39ed34b41a477d1370f843ba8a6f61a0c1d934"} Nov 03 01:02:14.338338 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:14.337882 6400 scope.go:122] "RemoveContainer" containerID="90064bb842370308cd7d6abb2679e7004d34351668345305eee7cb0085839a9f" Nov 03 01:02:14.347986 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:14.347960 6400 scope.go:122] "RemoveContainer" containerID="90064bb842370308cd7d6abb2679e7004d34351668345305eee7cb0085839a9f" Nov 03 01:02:14.348321 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:02:14.348302 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90064bb842370308cd7d6abb2679e7004d34351668345305eee7cb0085839a9f\": container with ID starting with 90064bb842370308cd7d6abb2679e7004d34351668345305eee7cb0085839a9f not found: ID does not exist" containerID="90064bb842370308cd7d6abb2679e7004d34351668345305eee7cb0085839a9f" Nov 03 01:02:14.348385 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:14.348330 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90064bb842370308cd7d6abb2679e7004d34351668345305eee7cb0085839a9f"} err="failed to get container status \"90064bb842370308cd7d6abb2679e7004d34351668345305eee7cb0085839a9f\": rpc error: code = NotFound desc = could not find container \"90064bb842370308cd7d6abb2679e7004d34351668345305eee7cb0085839a9f\": container with ID starting with 90064bb842370308cd7d6abb2679e7004d34351668345305eee7cb0085839a9f not found: ID does not exist" Nov 03 01:02:14.506755 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:14.506714 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh"] Nov 03 01:02:14.555909 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:14.555875 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f40c6c-6b669f4c54-fnvxh"] Nov 03 01:02:15.178692 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:15.178660 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a3a116-a7eb-4e1e-aea8-113107841624" path="/var/lib/kubelet/pods/87a3a116-a7eb-4e1e-aea8-113107841624/volumes" Nov 03 01:02:15.205954 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:15.205914 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:17.299413 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:17.297513 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:17.731064 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:17.730969 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:17.769168 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:17.769135 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw"] Nov 03 01:02:17.769403 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:17.769376 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" containerName="istio-proxy" containerID="cri-o://bbd38e8176ef1a958623975369bf98ea38c15ba5dc1089355defecd962c71dd3" gracePeriod=30 Nov 03 01:02:17.897891 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:17.897753 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:18.600789 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:18.600753 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:20.234265 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:20.234221 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:22.449360 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:22.449322 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:22.614673 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:22.614632 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:22.814530 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:22.814496 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 01:02:22.814670 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:22.814553 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 01:02:23.042304 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.042244 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 01:02:23.093277 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.093242 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-envoy\") pod \"e27d8659-f856-403f-a9eb-d86f17f24506\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " Nov 03 01:02:23.093449 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.093305 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-credential-socket\") pod \"e27d8659-f856-403f-a9eb-d86f17f24506\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " Nov 03 01:02:23.093449 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.093376 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e27d8659-f856-403f-a9eb-d86f17f24506-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27d8659-f856-403f-a9eb-d86f17f24506-istiod-ca-cert\") pod \"e27d8659-f856-403f-a9eb-d86f17f24506\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " Nov 03 01:02:23.093449 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.093437 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e27d8659-f856-403f-a9eb-d86f17f24506-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27d8659-f856-403f-a9eb-d86f17f24506-istio-podinfo\") pod \"e27d8659-f856-403f-a9eb-d86f17f24506\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " Nov 03 01:02:23.093649 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.093480 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-socket\") pod \"e27d8659-f856-403f-a9eb-d86f17f24506\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " Nov 03 01:02:23.093649 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.093507 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-certs\") pod \"e27d8659-f856-403f-a9eb-d86f17f24506\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " Nov 03 01:02:23.093649 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.093540 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-kube-api-access-qqx8d\" (UniqueName: \"kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-kube-api-access-qqx8d\") pod \"e27d8659-f856-403f-a9eb-d86f17f24506\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " Nov 03 01:02:23.093649 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.093610 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-data\") pod \"e27d8659-f856-403f-a9eb-d86f17f24506\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " Nov 03 01:02:23.093649 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.093648 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-istio-token\" (UniqueName: \"kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-istio-token\") pod \"e27d8659-f856-403f-a9eb-d86f17f24506\" (UID: \"e27d8659-f856-403f-a9eb-d86f17f24506\") " Nov 03 01:02:23.093915 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.093641 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-credential-socket" pod "e27d8659-f856-403f-a9eb-d86f17f24506" (UID: "e27d8659-f856-403f-a9eb-d86f17f24506"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:23.093915 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.093751 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27d8659-f856-403f-a9eb-d86f17f24506-istiod-ca-cert" pod "e27d8659-f856-403f-a9eb-d86f17f24506" (UID: "e27d8659-f856-403f-a9eb-d86f17f24506"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:23.094052 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.094024 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-certs" pod "e27d8659-f856-403f-a9eb-d86f17f24506" (UID: "e27d8659-f856-403f-a9eb-d86f17f24506"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:23.094052 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.094043 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-socket" pod "e27d8659-f856-403f-a9eb-d86f17f24506" (UID: "e27d8659-f856-403f-a9eb-d86f17f24506"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:23.094228 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.094198 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-data" pod "e27d8659-f856-403f-a9eb-d86f17f24506" (UID: "e27d8659-f856-403f-a9eb-d86f17f24506"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:23.094297 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.094274 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:23.094345 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.094298 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27d8659-f856-403f-a9eb-d86f17f24506-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:23.094345 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.094310 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:23.094345 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.094320 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:23.094345 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.094328 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:23.096792 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.096768 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-envoy" pod "e27d8659-f856-403f-a9eb-d86f17f24506" (UID: "e27d8659-f856-403f-a9eb-d86f17f24506"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:23.096792 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.096774 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e27d8659-f856-403f-a9eb-d86f17f24506-istio-podinfo" pod "e27d8659-f856-403f-a9eb-d86f17f24506" (UID: "e27d8659-f856-403f-a9eb-d86f17f24506"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:02:23.096939 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.096893 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-istio-token" pod "e27d8659-f856-403f-a9eb-d86f17f24506" (UID: "e27d8659-f856-403f-a9eb-d86f17f24506"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:23.096939 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.096898 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-kube-api-access-qqx8d" pod "e27d8659-f856-403f-a9eb-d86f17f24506" (UID: "e27d8659-f856-403f-a9eb-d86f17f24506"). InnerVolumeSpecName "kube-api-access-qqx8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:23.195669 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.195629 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27d8659-f856-403f-a9eb-d86f17f24506-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:23.195669 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.195671 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqx8d\" (UniqueName: \"kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-kube-api-access-qqx8d\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:23.195859 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.195688 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27d8659-f856-403f-a9eb-d86f17f24506-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:23.195859 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.195701 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27d8659-f856-403f-a9eb-d86f17f24506-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:23.392536 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.392445 6400 generic.go:356] "Generic (PLEG): container finished" podID="e27d8659-f856-403f-a9eb-d86f17f24506" containerID="bbd38e8176ef1a958623975369bf98ea38c15ba5dc1089355defecd962c71dd3" exitCode=0 Nov 03 01:02:23.392536 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.392496 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" Nov 03 01:02:23.392770 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.392541 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" event={"ID":"e27d8659-f856-403f-a9eb-d86f17f24506","Type":"ContainerDied","Data":"bbd38e8176ef1a958623975369bf98ea38c15ba5dc1089355defecd962c71dd3"} Nov 03 01:02:23.392770 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.392599 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw" event={"ID":"e27d8659-f856-403f-a9eb-d86f17f24506","Type":"ContainerDied","Data":"11b26b42e5db47442e92f737dc14da5de1c2f204d1e9aba93d340cea4cca2843"} Nov 03 01:02:23.392770 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.392626 6400 scope.go:122] "RemoveContainer" containerID="bbd38e8176ef1a958623975369bf98ea38c15ba5dc1089355defecd962c71dd3" Nov 03 01:02:23.402058 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.402034 6400 scope.go:122] "RemoveContainer" containerID="dc8ed8f15fe481d9f4ce9a6f4a210a69b6629d3b074c5158144f0a49a5df2434" Nov 03 01:02:23.413650 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.413628 6400 scope.go:122] "RemoveContainer" containerID="bbd38e8176ef1a958623975369bf98ea38c15ba5dc1089355defecd962c71dd3" Nov 03 01:02:23.413994 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:02:23.413975 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd38e8176ef1a958623975369bf98ea38c15ba5dc1089355defecd962c71dd3\": container with ID starting with bbd38e8176ef1a958623975369bf98ea38c15ba5dc1089355defecd962c71dd3 not found: ID does not exist" containerID="bbd38e8176ef1a958623975369bf98ea38c15ba5dc1089355defecd962c71dd3" Nov 03 01:02:23.414049 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.414002 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd38e8176ef1a958623975369bf98ea38c15ba5dc1089355defecd962c71dd3"} err="failed to get container status \"bbd38e8176ef1a958623975369bf98ea38c15ba5dc1089355defecd962c71dd3\": rpc error: code = NotFound desc = could not find container \"bbd38e8176ef1a958623975369bf98ea38c15ba5dc1089355defecd962c71dd3\": container with ID starting with bbd38e8176ef1a958623975369bf98ea38c15ba5dc1089355defecd962c71dd3 not found: ID does not exist" Nov 03 01:02:23.414049 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.414020 6400 scope.go:122] "RemoveContainer" containerID="dc8ed8f15fe481d9f4ce9a6f4a210a69b6629d3b074c5158144f0a49a5df2434" Nov 03 01:02:23.414249 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:02:23.414234 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc8ed8f15fe481d9f4ce9a6f4a210a69b6629d3b074c5158144f0a49a5df2434\": container with ID starting with dc8ed8f15fe481d9f4ce9a6f4a210a69b6629d3b074c5158144f0a49a5df2434 not found: ID does not exist" containerID="dc8ed8f15fe481d9f4ce9a6f4a210a69b6629d3b074c5158144f0a49a5df2434" Nov 03 01:02:23.414296 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.414253 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc8ed8f15fe481d9f4ce9a6f4a210a69b6629d3b074c5158144f0a49a5df2434"} err="failed to get container status \"dc8ed8f15fe481d9f4ce9a6f4a210a69b6629d3b074c5158144f0a49a5df2434\": rpc error: code = NotFound desc = could not find container \"dc8ed8f15fe481d9f4ce9a6f4a210a69b6629d3b074c5158144f0a49a5df2434\": container with ID starting with dc8ed8f15fe481d9f4ce9a6f4a210a69b6629d3b074c5158144f0a49a5df2434 not found: ID does not exist" Nov 03 01:02:23.555633 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.555600 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw"] Nov 03 01:02:23.558842 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:23.558803 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f40c6c-openshift-default-6967574b79-zwxhw"] Nov 03 01:02:24.381096 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.381062 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:24.388620 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.388594 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752"] Nov 03 01:02:24.388809 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.388790 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" podUID="c2e8b788-c97d-4739-821a-de697c19d429" containerName="maas-api" containerID="cri-o://2453ec56d4338dad95df746195755a7552c453fe078609c659d8fecd31113ea2" gracePeriod=30 Nov 03 01:02:24.588814 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.588779 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:24.692164 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.692136 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 01:02:24.832916 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.832878 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e8b788-c97d-4739-821a-de697c19d429-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e8b788-c97d-4739-821a-de697c19d429-openshift-service-ca\") pod \"c2e8b788-c97d-4739-821a-de697c19d429\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " Nov 03 01:02:24.833119 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.832995 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-metrics-certs\") pod \"c2e8b788-c97d-4739-821a-de697c19d429\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " Nov 03 01:02:24.834126 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.834069 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e8b788-c97d-4739-821a-de697c19d429-openshift-service-ca" pod "c2e8b788-c97d-4739-821a-de697c19d429" (UID: "c2e8b788-c97d-4739-821a-de697c19d429"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:24.872280 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.872242 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-metrics-certs" pod "c2e8b788-c97d-4739-821a-de697c19d429" (UID: "c2e8b788-c97d-4739-821a-de697c19d429"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:24.873883 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.873845 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e8b788-c97d-4739-821a-de697c19d429-kube-api-access-ccb6b\" (UniqueName: \"kubernetes.io/projected/c2e8b788-c97d-4739-821a-de697c19d429-kube-api-access-ccb6b\") pod \"c2e8b788-c97d-4739-821a-de697c19d429\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " Nov 03 01:02:24.874038 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.874020 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-maas-api-tls\") pod \"c2e8b788-c97d-4739-821a-de697c19d429\" (UID: \"c2e8b788-c97d-4739-821a-de697c19d429\") " Nov 03 01:02:24.874766 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.874744 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:24.874888 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.874773 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e8b788-c97d-4739-821a-de697c19d429-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:24.878114 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.877457 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-maas-api-tls" pod "c2e8b788-c97d-4739-821a-de697c19d429" (UID: "c2e8b788-c97d-4739-821a-de697c19d429"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:24.879534 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.879505 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e8b788-c97d-4739-821a-de697c19d429-kube-api-access-ccb6b" pod "c2e8b788-c97d-4739-821a-de697c19d429" (UID: "c2e8b788-c97d-4739-821a-de697c19d429"). InnerVolumeSpecName "kube-api-access-ccb6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:24.976246 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.976150 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccb6b\" (UniqueName: \"kubernetes.io/projected/c2e8b788-c97d-4739-821a-de697c19d429-kube-api-access-ccb6b\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:24.976246 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:24.976194 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e8b788-c97d-4739-821a-de697c19d429-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:25.115258 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:25.115216 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:25.180351 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:25.180317 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" path="/var/lib/kubelet/pods/e27d8659-f856-403f-a9eb-d86f17f24506/volumes" Nov 03 01:02:25.373041 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:25.372998 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:25.407043 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:25.407004 6400 generic.go:356] "Generic (PLEG): container finished" podID="c2e8b788-c97d-4739-821a-de697c19d429" containerID="2453ec56d4338dad95df746195755a7552c453fe078609c659d8fecd31113ea2" exitCode=0 Nov 03 01:02:25.407228 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:25.407081 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" Nov 03 01:02:25.407228 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:25.407098 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" event={"ID":"c2e8b788-c97d-4739-821a-de697c19d429","Type":"ContainerDied","Data":"2453ec56d4338dad95df746195755a7552c453fe078609c659d8fecd31113ea2"} Nov 03 01:02:25.407228 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:25.407168 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752" event={"ID":"c2e8b788-c97d-4739-821a-de697c19d429","Type":"ContainerDied","Data":"e4cfd74ac29ac0268dd748d8470880e59e72c73d5f0162fcc1f87bcd32e087de"} Nov 03 01:02:25.407228 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:25.407198 6400 scope.go:122] "RemoveContainer" containerID="2453ec56d4338dad95df746195755a7552c453fe078609c659d8fecd31113ea2" Nov 03 01:02:25.417759 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:25.417737 6400 scope.go:122] "RemoveContainer" containerID="2453ec56d4338dad95df746195755a7552c453fe078609c659d8fecd31113ea2" Nov 03 01:02:25.418061 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:02:25.418043 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2453ec56d4338dad95df746195755a7552c453fe078609c659d8fecd31113ea2\": container with ID starting with 2453ec56d4338dad95df746195755a7552c453fe078609c659d8fecd31113ea2 not found: ID does not exist" containerID="2453ec56d4338dad95df746195755a7552c453fe078609c659d8fecd31113ea2" Nov 03 01:02:25.418125 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:25.418072 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2453ec56d4338dad95df746195755a7552c453fe078609c659d8fecd31113ea2"} err="failed to get container status \"2453ec56d4338dad95df746195755a7552c453fe078609c659d8fecd31113ea2\": rpc error: code = NotFound desc = could not find container \"2453ec56d4338dad95df746195755a7552c453fe078609c659d8fecd31113ea2\": container with ID starting with 2453ec56d4338dad95df746195755a7552c453fe078609c659d8fecd31113ea2 not found: ID does not exist" Nov 03 01:02:25.575764 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:25.575734 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752"] Nov 03 01:02:25.581170 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:25.581135 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3b1d36-6b7c8499fd-s5752"] Nov 03 01:02:26.605834 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:26.605788 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:27.152941 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:02:27.152905 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:02:27.178907 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:27.178878 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e8b788-c97d-4739-821a-de697c19d429" path="/var/lib/kubelet/pods/c2e8b788-c97d-4739-821a-de697c19d429/volumes" Nov 03 01:02:30.100668 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:30.100628 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:30.299629 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:30.299597 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:31.629313 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:31.629262 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:31.787285 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:31.787235 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:31.942077 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:31.941978 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:32.449446 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:32.449406 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj"] Nov 03 01:02:32.449627 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:32.449604 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" containerID="cri-o://7e334ae080fa5056225586578d2b4f46a65de809a3ab78f40003c6256c082f48" gracePeriod=30 Nov 03 01:02:32.471616 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:32.471577 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:34.234505 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:34.234470 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:34.398039 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:34.397996 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:36.459158 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:36.459115 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:02:36.459561 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:36.459189 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:02:36.860680 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:36.860646 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:37.706131 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.706104 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 01:02:37.834633 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.834603 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73e7ca86-4542-4352-889a-bc6181318baa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e7ca86-4542-4352-889a-bc6181318baa-istio-podinfo\") pod \"73e7ca86-4542-4352-889a-bc6181318baa\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " Nov 03 01:02:37.834810 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.834679 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-certs\") pod \"73e7ca86-4542-4352-889a-bc6181318baa\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " Nov 03 01:02:37.834810 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.834739 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-kube-api-access-tfm6z\" (UniqueName: \"kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-kube-api-access-tfm6z\") pod \"73e7ca86-4542-4352-889a-bc6181318baa\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " Nov 03 01:02:37.834810 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.834787 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-istio-token\" (UniqueName: \"kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-istio-token\") pod \"73e7ca86-4542-4352-889a-bc6181318baa\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " Nov 03 01:02:37.835007 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.834877 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e7ca86-4542-4352-889a-bc6181318baa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e7ca86-4542-4352-889a-bc6181318baa-istiod-ca-cert\") pod \"73e7ca86-4542-4352-889a-bc6181318baa\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " Nov 03 01:02:37.835007 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.834901 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-credential-socket\") pod \"73e7ca86-4542-4352-889a-bc6181318baa\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " Nov 03 01:02:37.835007 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.834921 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-socket\") pod \"73e7ca86-4542-4352-889a-bc6181318baa\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " Nov 03 01:02:37.835007 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.835004 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-data\") pod \"73e7ca86-4542-4352-889a-bc6181318baa\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " Nov 03 01:02:37.835224 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.835032 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-envoy\") pod \"73e7ca86-4542-4352-889a-bc6181318baa\" (UID: \"73e7ca86-4542-4352-889a-bc6181318baa\") " Nov 03 01:02:37.835224 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.835041 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-certs" pod "73e7ca86-4542-4352-889a-bc6181318baa" (UID: "73e7ca86-4542-4352-889a-bc6181318baa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:37.835224 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.835179 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-credential-socket" pod "73e7ca86-4542-4352-889a-bc6181318baa" (UID: "73e7ca86-4542-4352-889a-bc6181318baa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:37.835398 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.835257 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-socket" pod "73e7ca86-4542-4352-889a-bc6181318baa" (UID: "73e7ca86-4542-4352-889a-bc6181318baa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:37.835398 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.835270 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e7ca86-4542-4352-889a-bc6181318baa-istiod-ca-cert" pod "73e7ca86-4542-4352-889a-bc6181318baa" (UID: "73e7ca86-4542-4352-889a-bc6181318baa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:37.835678 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.835396 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-data" pod "73e7ca86-4542-4352-889a-bc6181318baa" (UID: "73e7ca86-4542-4352-889a-bc6181318baa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:37.835678 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.835587 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e7ca86-4542-4352-889a-bc6181318baa-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.835678 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.835603 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.835678 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.835612 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.835678 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.835623 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.835678 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.835637 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.837038 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.837013 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73e7ca86-4542-4352-889a-bc6181318baa-istio-podinfo" pod "73e7ca86-4542-4352-889a-bc6181318baa" (UID: "73e7ca86-4542-4352-889a-bc6181318baa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:02:37.837211 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.837195 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-kube-api-access-tfm6z" pod "73e7ca86-4542-4352-889a-bc6181318baa" (UID: "73e7ca86-4542-4352-889a-bc6181318baa"). InnerVolumeSpecName "kube-api-access-tfm6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:37.837267 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.837215 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-istio-token" pod "73e7ca86-4542-4352-889a-bc6181318baa" (UID: "73e7ca86-4542-4352-889a-bc6181318baa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:37.837608 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.837592 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-envoy" pod "73e7ca86-4542-4352-889a-bc6181318baa" (UID: "73e7ca86-4542-4352-889a-bc6181318baa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:37.936981 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.936944 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e7ca86-4542-4352-889a-bc6181318baa-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.936981 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.936980 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e7ca86-4542-4352-889a-bc6181318baa-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.937187 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.936993 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfm6z\" (UniqueName: \"kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-kube-api-access-tfm6z\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.937187 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:37.937005 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e7ca86-4542-4352-889a-bc6181318baa-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:38.481874 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:38.481813 6400 generic.go:356] "Generic (PLEG): container finished" podID="73e7ca86-4542-4352-889a-bc6181318baa" containerID="7e334ae080fa5056225586578d2b4f46a65de809a3ab78f40003c6256c082f48" exitCode=0 Nov 03 01:02:38.482072 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:38.481911 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" Nov 03 01:02:38.482072 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:38.481923 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" event={"ID":"73e7ca86-4542-4352-889a-bc6181318baa","Type":"ContainerDied","Data":"7e334ae080fa5056225586578d2b4f46a65de809a3ab78f40003c6256c082f48"} Nov 03 01:02:38.482072 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:38.481982 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj" event={"ID":"73e7ca86-4542-4352-889a-bc6181318baa","Type":"ContainerDied","Data":"3f4954f6a3bae63eb56b7211736b61c4bd391f2c67a3f2a02a3f79a20c80aad0"} Nov 03 01:02:38.482072 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:38.482011 6400 scope.go:122] "RemoveContainer" containerID="7e334ae080fa5056225586578d2b4f46a65de809a3ab78f40003c6256c082f48" Nov 03 01:02:38.492719 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:38.492697 6400 scope.go:122] "RemoveContainer" containerID="d00cb8c36ec37c3d173e7ef91071d3e1a522e2881fd40efdee111185f7aa5ba1" Nov 03 01:02:38.501792 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:38.501773 6400 scope.go:122] "RemoveContainer" containerID="7e334ae080fa5056225586578d2b4f46a65de809a3ab78f40003c6256c082f48" Nov 03 01:02:38.502134 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:02:38.502109 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e334ae080fa5056225586578d2b4f46a65de809a3ab78f40003c6256c082f48\": container with ID starting with 7e334ae080fa5056225586578d2b4f46a65de809a3ab78f40003c6256c082f48 not found: ID does not exist" containerID="7e334ae080fa5056225586578d2b4f46a65de809a3ab78f40003c6256c082f48" Nov 03 01:02:38.502198 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:38.502146 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e334ae080fa5056225586578d2b4f46a65de809a3ab78f40003c6256c082f48"} err="failed to get container status \"7e334ae080fa5056225586578d2b4f46a65de809a3ab78f40003c6256c082f48\": rpc error: code = NotFound desc = could not find container \"7e334ae080fa5056225586578d2b4f46a65de809a3ab78f40003c6256c082f48\": container with ID starting with 7e334ae080fa5056225586578d2b4f46a65de809a3ab78f40003c6256c082f48 not found: ID does not exist" Nov 03 01:02:38.502198 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:38.502169 6400 scope.go:122] "RemoveContainer" containerID="d00cb8c36ec37c3d173e7ef91071d3e1a522e2881fd40efdee111185f7aa5ba1" Nov 03 01:02:38.502407 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:02:38.502389 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00cb8c36ec37c3d173e7ef91071d3e1a522e2881fd40efdee111185f7aa5ba1\": container with ID starting with d00cb8c36ec37c3d173e7ef91071d3e1a522e2881fd40efdee111185f7aa5ba1 not found: ID does not exist" containerID="d00cb8c36ec37c3d173e7ef91071d3e1a522e2881fd40efdee111185f7aa5ba1" Nov 03 01:02:38.502470 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:38.502419 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00cb8c36ec37c3d173e7ef91071d3e1a522e2881fd40efdee111185f7aa5ba1"} err="failed to get container status \"d00cb8c36ec37c3d173e7ef91071d3e1a522e2881fd40efdee111185f7aa5ba1\": rpc error: code = NotFound desc = could not find container \"d00cb8c36ec37c3d173e7ef91071d3e1a522e2881fd40efdee111185f7aa5ba1\": container with ID starting with d00cb8c36ec37c3d173e7ef91071d3e1a522e2881fd40efdee111185f7aa5ba1 not found: ID does not exist" Nov 03 01:02:38.666763 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:38.666726 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj"] Nov 03 01:02:38.671195 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:38.671173 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3b1d36-openshift-default-68b684c967-wq9bj"] Nov 03 01:02:39.178942 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:39.178907 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e7ca86-4542-4352-889a-bc6181318baa" path="/var/lib/kubelet/pods/73e7ca86-4542-4352-889a-bc6181318baa/volumes" Nov 03 01:02:42.006423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:42.006385 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:46.652614 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.652577 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb"] Nov 03 01:02:46.653262 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653241 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" containerName="cleanup" Nov 03 01:02:46.653348 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653263 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da8cdf3-0d59-48ca-ad28-6a85adac1544" containerName="cleanup" Nov 03 01:02:46.653348 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653279 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" containerName="istio-proxy" Nov 03 01:02:46.653348 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653286 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" containerName="istio-proxy" Nov 03 01:02:46.653348 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653307 6400 cpu_manager.go:414] "containerMap: removing container" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" Nov 03 01:02:46.653348 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653314 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" Nov 03 01:02:46.653348 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653333 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c2e8b788-c97d-4739-821a-de697c19d429" containerName="maas-api" Nov 03 01:02:46.653348 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653340 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e8b788-c97d-4739-821a-de697c19d429" containerName="maas-api" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653353 6400 cpu_manager.go:414] "containerMap: removing container" podUID="780105c6-b952-4193-b287-9e93040ea9e6" containerName="cleanup" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653360 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780105c6-b952-4193-b287-9e93040ea9e6" containerName="cleanup" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653386 6400 cpu_manager.go:414] "containerMap: removing container" podUID="87a3a116-a7eb-4e1e-aea8-113107841624" containerName="maas-api" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653394 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a3a116-a7eb-4e1e-aea8-113107841624" containerName="maas-api" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653410 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" containerName="cleanup" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653417 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" containerName="cleanup" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653444 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" containerName="istio-proxy" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653451 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" containerName="istio-proxy" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653466 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" containerName="cleanup" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653473 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" containerName="cleanup" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653584 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a3a116-a7eb-4e1e-aea8-113107841624" containerName="maas-api" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653600 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653613 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f55000e-0fcb-4045-a07b-7f89fe9f20c3" containerName="cleanup" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653622 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" containerName="istio-proxy" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653632 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f2f5d3-8ed3-417f-b35f-81e082b77aa2" containerName="cleanup" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653642 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="780105c6-b952-4193-b287-9e93040ea9e6" containerName="cleanup" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653650 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f017544e-4cbe-4f21-abdc-70b6c6ab8981" containerName="cleanup" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653662 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27d8659-f856-403f-a9eb-d86f17f24506" containerName="istio-proxy" Nov 03 01:02:46.653702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.653673 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e8b788-c97d-4739-821a-de697c19d429" containerName="maas-api" Nov 03 01:02:46.658439 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.658419 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:46.703552 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.703522 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb"] Nov 03 01:02:46.737321 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.737291 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:46.737462 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.737361 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwpsw\" (UniqueName: \"kubernetes.io/projected/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-kube-api-access-rwpsw\") pod \"maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:46.737509 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.737473 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-metrics-certs\") pod \"maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:46.737509 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.737503 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:46.751035 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.751011 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-cac9f3-serving-cert\"" Nov 03 01:02:46.838349 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.838307 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:46.838560 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.838415 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwpsw\" (UniqueName: \"kubernetes.io/projected/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-kube-api-access-rwpsw\") pod \"maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:46.838560 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:02:46.838452 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-serving-cert: secret "maas-api-e2e-extmod-a-cac9f3-serving-cert" not found Nov 03 01:02:46.838560 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.838493 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-metrics-certs\") pod \"maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:46.838560 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:02:46.838510 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-maas-api-tls podName:c3a73031-e0ee-45a7-ab4d-8a947f9812a2 nodeName:}" failed. No retries permitted until 2026-11-03 01:02:47.338495659 +0000 UTC m=+3414.289435815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-maas-api-tls") pod "maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" (UID: "c3a73031-e0ee-45a7-ab4d-8a947f9812a2") : secret "maas-api-e2e-extmod-a-cac9f3-serving-cert" not found Nov 03 01:02:46.838560 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.838550 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:46.839228 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.839198 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:46.841059 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.841039 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-metrics-certs\") pod \"maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:46.859127 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.859099 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:46.908427 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:46.908319 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwpsw\" (UniqueName: \"kubernetes.io/projected/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-kube-api-access-rwpsw\") pod \"maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:47.021346 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:47.021305 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:47.174404 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:47.174326 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:47.343087 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:47.343047 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:47.345846 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:47.345789 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:47.571527 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:47.571498 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:47.946048 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:47.946025 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb"] Nov 03 01:02:47.948175 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:02:47.948148 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a73031_e0ee_45a7_ab4d_8a947f9812a2.slice/crio-254410fa3f3424e51bfbbb915e2e2495aabea6ee4fc6d9df862df803f8a1677d WatchSource:0}: Error finding container 254410fa3f3424e51bfbbb915e2e2495aabea6ee4fc6d9df862df803f8a1677d: Status 404 returned error can't find the container with id 254410fa3f3424e51bfbbb915e2e2495aabea6ee4fc6d9df862df803f8a1677d Nov 03 01:02:47.949746 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:47.949727 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:02:48.540073 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:48.540044 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" event={"ID":"c3a73031-e0ee-45a7-ab4d-8a947f9812a2","Type":"ContainerStarted","Data":"254410fa3f3424e51bfbbb915e2e2495aabea6ee4fc6d9df862df803f8a1677d"} Nov 03 01:02:49.547076 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:49.547043 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" event={"ID":"c3a73031-e0ee-45a7-ab4d-8a947f9812a2","Type":"ContainerStarted","Data":"2affa1d592124b24c69422e0ae8e21be514e9be1041fe33052a9a375c9635108"} Nov 03 01:02:49.547491 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:49.547204 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:49.661218 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:49.661155 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" podStartSLOduration=2.724911821 podStartE2EDuration="3.661137917s" totalImagesPullingTime="936.226096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:49.660143929 +0000 UTC m=+3416.611084088" watchObservedRunningTime="2026-11-03 01:02:49.661137917 +0000 UTC m=+3416.612078080" Nov 03 01:02:51.101421 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:51.101383 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:52.790291 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:52.790253 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:52.947868 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:52.947831 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:53.279380 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:53.279288 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:55.558049 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:55.558023 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:02:55.784929 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:55.784897 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:02:56.458191 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:56.458157 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:56.683883 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:56.683845 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:57.154664 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:02:57.154623 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:02:57.873932 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:02:57.873899 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:00.873403 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:00.873366 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:01.041339 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:01.041304 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:02.723247 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:02.723209 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:07.492768 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:07.492722 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:12.155639 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:12.155609 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:16.127423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.127387 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj"] Nov 03 01:03:16.127952 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.127611 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" podUID="8e8005c1-0893-45e6-82d5-3148021e8369" containerName="maas-api" containerID="cri-o://73383b44c40d339bd53f2f757569f9b2063de17d9eb6e7b6d6703da88e505939" gracePeriod=30 Nov 03 01:03:16.266280 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.266246 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:16.437623 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.437599 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:03:16.576142 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.576110 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-metrics-certs\") pod \"8e8005c1-0893-45e6-82d5-3148021e8369\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " Nov 03 01:03:16.576328 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.576168 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-maas-api-tls\") pod \"8e8005c1-0893-45e6-82d5-3148021e8369\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " Nov 03 01:03:16.576328 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.576277 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8005c1-0893-45e6-82d5-3148021e8369-kube-api-access-nhg5k\" (UniqueName: \"kubernetes.io/projected/8e8005c1-0893-45e6-82d5-3148021e8369-kube-api-access-nhg5k\") pod \"8e8005c1-0893-45e6-82d5-3148021e8369\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " Nov 03 01:03:16.576461 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.576351 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e8005c1-0893-45e6-82d5-3148021e8369-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8005c1-0893-45e6-82d5-3148021e8369-openshift-service-ca\") pod \"8e8005c1-0893-45e6-82d5-3148021e8369\" (UID: \"8e8005c1-0893-45e6-82d5-3148021e8369\") " Nov 03 01:03:16.577131 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.577018 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8005c1-0893-45e6-82d5-3148021e8369-openshift-service-ca" pod "8e8005c1-0893-45e6-82d5-3148021e8369" (UID: "8e8005c1-0893-45e6-82d5-3148021e8369"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:16.578587 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.578550 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-maas-api-tls" pod "8e8005c1-0893-45e6-82d5-3148021e8369" (UID: "8e8005c1-0893-45e6-82d5-3148021e8369"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:16.578867 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.578765 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8005c1-0893-45e6-82d5-3148021e8369-kube-api-access-nhg5k" pod "8e8005c1-0893-45e6-82d5-3148021e8369" (UID: "8e8005c1-0893-45e6-82d5-3148021e8369"). InnerVolumeSpecName "kube-api-access-nhg5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:16.580084 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.580062 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-metrics-certs" pod "8e8005c1-0893-45e6-82d5-3148021e8369" (UID: "8e8005c1-0893-45e6-82d5-3148021e8369"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:16.685526 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.684858 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:16.685526 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.684893 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8005c1-0893-45e6-82d5-3148021e8369-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:16.685526 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.684907 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhg5k\" (UniqueName: \"kubernetes.io/projected/8e8005c1-0893-45e6-82d5-3148021e8369-kube-api-access-nhg5k\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:16.685526 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.684923 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8005c1-0893-45e6-82d5-3148021e8369-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:16.702965 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.702927 6400 generic.go:356] "Generic (PLEG): container finished" podID="8e8005c1-0893-45e6-82d5-3148021e8369" containerID="73383b44c40d339bd53f2f757569f9b2063de17d9eb6e7b6d6703da88e505939" exitCode=0 Nov 03 01:03:16.703189 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.702983 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" event={"ID":"8e8005c1-0893-45e6-82d5-3148021e8369","Type":"ContainerDied","Data":"73383b44c40d339bd53f2f757569f9b2063de17d9eb6e7b6d6703da88e505939"} Nov 03 01:03:16.703189 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.703026 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" event={"ID":"8e8005c1-0893-45e6-82d5-3148021e8369","Type":"ContainerDied","Data":"73077f8967731e56538d233800b1598fdfb1643514e072a8895999408c30236b"} Nov 03 01:03:16.703189 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.703052 6400 scope.go:122] "RemoveContainer" containerID="73383b44c40d339bd53f2f757569f9b2063de17d9eb6e7b6d6703da88e505939" Nov 03 01:03:16.703189 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.702988 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj" Nov 03 01:03:16.716971 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.716923 6400 scope.go:122] "RemoveContainer" containerID="73383b44c40d339bd53f2f757569f9b2063de17d9eb6e7b6d6703da88e505939" Nov 03 01:03:16.717189 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:03:16.717167 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73383b44c40d339bd53f2f757569f9b2063de17d9eb6e7b6d6703da88e505939\": container with ID starting with 73383b44c40d339bd53f2f757569f9b2063de17d9eb6e7b6d6703da88e505939 not found: ID does not exist" containerID="73383b44c40d339bd53f2f757569f9b2063de17d9eb6e7b6d6703da88e505939" Nov 03 01:03:16.717267 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.717202 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73383b44c40d339bd53f2f757569f9b2063de17d9eb6e7b6d6703da88e505939"} err="failed to get container status \"73383b44c40d339bd53f2f757569f9b2063de17d9eb6e7b6d6703da88e505939\": rpc error: code = NotFound desc = could not find container \"73383b44c40d339bd53f2f757569f9b2063de17d9eb6e7b6d6703da88e505939\": container with ID starting with 73383b44c40d339bd53f2f757569f9b2063de17d9eb6e7b6d6703da88e505939 not found: ID does not exist" Nov 03 01:03:16.866010 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.865973 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj"] Nov 03 01:03:16.871854 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:16.871832 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-642e6a-d64d66b8c-jjqjj"] Nov 03 01:03:17.179065 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:17.179032 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8005c1-0893-45e6-82d5-3148021e8369" path="/var/lib/kubelet/pods/8e8005c1-0893-45e6-82d5-3148021e8369/volumes" Nov 03 01:03:17.971552 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:17.971505 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:18.168099 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:18.168060 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:22.713729 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:22.713688 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:24.361839 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:24.361777 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:24.569624 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:24.569595 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:25.935264 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:25.935226 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:26.434435 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:26.434397 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx"] Nov 03 01:03:26.434769 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:26.434736 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" podUID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerName="istio-proxy" containerID="cri-o://b6314d9664b8eb08ac4d6b2c9503b3c23ccac4d61b528bdac8b03a9216a13099" gracePeriod=30 Nov 03 01:03:26.448748 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:26.448714 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:27.152451 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:03:27.152411 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:03:27.614397 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:27.614359 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:28.236602 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:28.236563 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:28.451193 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:28.451156 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:31.702745 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.702681 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 01:03:31.758737 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.758703 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-kube-api-access-rchwg\" (UniqueName: \"kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-kube-api-access-rchwg\") pod \"bdae0d76-2372-4d89-89e2-84e69f787b99\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " Nov 03 01:03:31.758971 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.758746 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-envoy\") pod \"bdae0d76-2372-4d89-89e2-84e69f787b99\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " Nov 03 01:03:31.758971 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.758781 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-credential-socket\") pod \"bdae0d76-2372-4d89-89e2-84e69f787b99\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " Nov 03 01:03:31.758971 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.758874 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-certs\") pod \"bdae0d76-2372-4d89-89e2-84e69f787b99\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " Nov 03 01:03:31.758971 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.758952 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-podinfo\") pod \"bdae0d76-2372-4d89-89e2-84e69f787b99\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " Nov 03 01:03:31.759197 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759003 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdae0d76-2372-4d89-89e2-84e69f787b99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdae0d76-2372-4d89-89e2-84e69f787b99-istiod-ca-cert\") pod \"bdae0d76-2372-4d89-89e2-84e69f787b99\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " Nov 03 01:03:31.759197 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759047 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-token\" (UniqueName: \"kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-token\") pod \"bdae0d76-2372-4d89-89e2-84e69f787b99\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " Nov 03 01:03:31.759197 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759083 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-socket\") pod \"bdae0d76-2372-4d89-89e2-84e69f787b99\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " Nov 03 01:03:31.759197 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759131 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-credential-socket" pod "bdae0d76-2372-4d89-89e2-84e69f787b99" (UID: "bdae0d76-2372-4d89-89e2-84e69f787b99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:31.759197 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759183 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-data\") pod \"bdae0d76-2372-4d89-89e2-84e69f787b99\" (UID: \"bdae0d76-2372-4d89-89e2-84e69f787b99\") " Nov 03 01:03:31.759466 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759206 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-certs" pod "bdae0d76-2372-4d89-89e2-84e69f787b99" (UID: "bdae0d76-2372-4d89-89e2-84e69f787b99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:31.759466 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759444 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-socket" pod "bdae0d76-2372-4d89-89e2-84e69f787b99" (UID: "bdae0d76-2372-4d89-89e2-84e69f787b99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:31.759739 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759485 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdae0d76-2372-4d89-89e2-84e69f787b99-istiod-ca-cert" pod "bdae0d76-2372-4d89-89e2-84e69f787b99" (UID: "bdae0d76-2372-4d89-89e2-84e69f787b99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:31.759739 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759641 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-data" pod "bdae0d76-2372-4d89-89e2-84e69f787b99" (UID: "bdae0d76-2372-4d89-89e2-84e69f787b99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:31.759931 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759911 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:31.760010 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759935 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:31.760010 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759949 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:31.760010 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759959 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdae0d76-2372-4d89-89e2-84e69f787b99-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:31.760010 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.759967 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:31.761262 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.761242 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-podinfo" pod "bdae0d76-2372-4d89-89e2-84e69f787b99" (UID: "bdae0d76-2372-4d89-89e2-84e69f787b99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:03:31.761381 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.761283 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-envoy" pod "bdae0d76-2372-4d89-89e2-84e69f787b99" (UID: "bdae0d76-2372-4d89-89e2-84e69f787b99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:31.761606 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.761587 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-token" pod "bdae0d76-2372-4d89-89e2-84e69f787b99" (UID: "bdae0d76-2372-4d89-89e2-84e69f787b99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:31.761666 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.761598 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-kube-api-access-rchwg" pod "bdae0d76-2372-4d89-89e2-84e69f787b99" (UID: "bdae0d76-2372-4d89-89e2-84e69f787b99"). InnerVolumeSpecName "kube-api-access-rchwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:31.796505 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.796473 6400 generic.go:356] "Generic (PLEG): container finished" podID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerID="b6314d9664b8eb08ac4d6b2c9503b3c23ccac4d61b528bdac8b03a9216a13099" exitCode=0 Nov 03 01:03:31.796654 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.796514 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" event={"ID":"bdae0d76-2372-4d89-89e2-84e69f787b99","Type":"ContainerDied","Data":"b6314d9664b8eb08ac4d6b2c9503b3c23ccac4d61b528bdac8b03a9216a13099"} Nov 03 01:03:31.796654 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.796552 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" Nov 03 01:03:31.796654 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.796585 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx" event={"ID":"bdae0d76-2372-4d89-89e2-84e69f787b99","Type":"ContainerDied","Data":"fc930b2689b6719a80bc0eb5c9085f21ff05c08b9819ae59b645e1f1578bc546"} Nov 03 01:03:31.796654 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.796615 6400 scope.go:122] "RemoveContainer" containerID="b6314d9664b8eb08ac4d6b2c9503b3c23ccac4d61b528bdac8b03a9216a13099" Nov 03 01:03:31.810752 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.810716 6400 scope.go:122] "RemoveContainer" containerID="df6735d6ecd70913368ee05b400ad16ec3eed9791ea809484bce14479d17b092" Nov 03 01:03:31.819443 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.819425 6400 scope.go:122] "RemoveContainer" containerID="b6314d9664b8eb08ac4d6b2c9503b3c23ccac4d61b528bdac8b03a9216a13099" Nov 03 01:03:31.819703 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:03:31.819671 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6314d9664b8eb08ac4d6b2c9503b3c23ccac4d61b528bdac8b03a9216a13099\": container with ID starting with b6314d9664b8eb08ac4d6b2c9503b3c23ccac4d61b528bdac8b03a9216a13099 not found: ID does not exist" containerID="b6314d9664b8eb08ac4d6b2c9503b3c23ccac4d61b528bdac8b03a9216a13099" Nov 03 01:03:31.819749 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.819716 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6314d9664b8eb08ac4d6b2c9503b3c23ccac4d61b528bdac8b03a9216a13099"} err="failed to get container status \"b6314d9664b8eb08ac4d6b2c9503b3c23ccac4d61b528bdac8b03a9216a13099\": rpc error: code = NotFound desc = could not find container \"b6314d9664b8eb08ac4d6b2c9503b3c23ccac4d61b528bdac8b03a9216a13099\": container with ID starting with b6314d9664b8eb08ac4d6b2c9503b3c23ccac4d61b528bdac8b03a9216a13099 not found: ID does not exist" Nov 03 01:03:31.819749 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.819736 6400 scope.go:122] "RemoveContainer" containerID="df6735d6ecd70913368ee05b400ad16ec3eed9791ea809484bce14479d17b092" Nov 03 01:03:31.820037 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:03:31.820019 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df6735d6ecd70913368ee05b400ad16ec3eed9791ea809484bce14479d17b092\": container with ID starting with df6735d6ecd70913368ee05b400ad16ec3eed9791ea809484bce14479d17b092 not found: ID does not exist" containerID="df6735d6ecd70913368ee05b400ad16ec3eed9791ea809484bce14479d17b092" Nov 03 01:03:31.820102 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.820047 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df6735d6ecd70913368ee05b400ad16ec3eed9791ea809484bce14479d17b092"} err="failed to get container status \"df6735d6ecd70913368ee05b400ad16ec3eed9791ea809484bce14479d17b092\": rpc error: code = NotFound desc = could not find container \"df6735d6ecd70913368ee05b400ad16ec3eed9791ea809484bce14479d17b092\": container with ID starting with df6735d6ecd70913368ee05b400ad16ec3eed9791ea809484bce14479d17b092 not found: ID does not exist" Nov 03 01:03:31.861045 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.861014 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rchwg\" (UniqueName: \"kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-kube-api-access-rchwg\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:31.861045 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.861044 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:31.861045 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.861053 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:31.861261 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.861062 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdae0d76-2372-4d89-89e2-84e69f787b99-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:31.962650 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.962563 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx"] Nov 03 01:03:31.967514 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:31.967487 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-642e6a-openshift-default-6677f55bfc-pzgxx"] Nov 03 01:03:32.624406 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:32.624373 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:33.178761 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:33.178728 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdae0d76-2372-4d89-89e2-84e69f787b99" path="/var/lib/kubelet/pods/bdae0d76-2372-4d89-89e2-84e69f787b99/volumes" Nov 03 01:03:35.410248 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:35.410210 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:35.578796 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:35.578746 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:37.354387 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:37.354355 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:39.039989 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:39.039948 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:39.620417 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:39.620375 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:39.702456 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:39.702372 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:03:39.806973 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:39.806934 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:42.139197 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:42.139160 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:43.676482 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:43.676439 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:43.853123 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:43.853085 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:45.762561 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.762525 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25"] Nov 03 01:03:45.762980 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.762972 6400 cpu_manager.go:414] "containerMap: removing container" podUID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerName="istio-proxy" Nov 03 01:03:45.763028 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.762984 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerName="istio-proxy" Nov 03 01:03:45.763063 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.763029 6400 cpu_manager.go:414] "containerMap: removing container" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" Nov 03 01:03:45.763063 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.763035 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" Nov 03 01:03:45.763063 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.763045 6400 cpu_manager.go:414] "containerMap: removing container" podUID="8e8005c1-0893-45e6-82d5-3148021e8369" containerName="maas-api" Nov 03 01:03:45.763063 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.763050 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8005c1-0893-45e6-82d5-3148021e8369" containerName="maas-api" Nov 03 01:03:45.763192 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.763117 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerName="istio-proxy" Nov 03 01:03:45.763192 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.763128 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8005c1-0893-45e6-82d5-3148021e8369" containerName="maas-api" Nov 03 01:03:45.763192 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.763133 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerName="istio-proxy" Nov 03 01:03:45.763192 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.763141 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e7ca86-4542-4352-889a-bc6181318baa" containerName="istio-proxy" Nov 03 01:03:45.767225 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.767203 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:45.812175 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.812135 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr"] Nov 03 01:03:45.812591 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.812578 6400 cpu_manager.go:414] "containerMap: removing container" podUID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerName="istio-proxy" Nov 03 01:03:45.812643 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.812592 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdae0d76-2372-4d89-89e2-84e69f787b99" containerName="istio-proxy" Nov 03 01:03:45.815672 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.815655 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:45.867815 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.867787 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25"] Nov 03 01:03:45.923264 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923228 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:45.923472 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923278 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:45.923472 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923370 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:45.923472 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923433 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:45.923472 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923457 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:45.923695 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923481 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ce4a34-1291-40bc-b3de-95ad93828326-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:45.923695 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923501 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17cab1-0794-408d-9d68-bd68708e3d39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:45.923695 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923593 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smdh6\" (UniqueName: \"kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-kube-api-access-smdh6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:45.923695 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923634 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:45.923695 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923653 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:45.923983 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923733 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:45.923983 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923758 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:45.923983 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923788 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ce4a34-1291-40bc-b3de-95ad93828326-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:45.923983 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923853 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:45.923983 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923901 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:45.923983 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923917 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:45.923983 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.923945 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bn5g\" (UniqueName: \"kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-kube-api-access-8bn5g\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:45.924356 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.924006 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:45.931260 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:45.931235 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr"] Nov 03 01:03:46.025143 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025050 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.025143 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025105 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.025143 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025133 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.025438 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025156 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ce4a34-1291-40bc-b3de-95ad93828326-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.025438 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025184 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17cab1-0794-408d-9d68-bd68708e3d39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.025438 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025223 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smdh6\" (UniqueName: \"kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-kube-api-access-smdh6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.025438 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025269 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.025438 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025291 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.025438 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025322 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.025765 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025516 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.025765 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025564 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.025765 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025575 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.025765 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025629 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ce4a34-1291-40bc-b3de-95ad93828326-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.025765 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025666 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.025765 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025714 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.025765 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025737 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.026181 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025785 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bn5g\" (UniqueName: \"kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-kube-api-access-8bn5g\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.026181 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025860 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.026181 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025917 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.026181 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025897 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.026181 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025946 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.026181 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025986 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.026181 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025998 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.026181 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.025628 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.026978 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.026270 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.026978 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.026308 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ce4a34-1291-40bc-b3de-95ad93828326-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.026978 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.026643 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.027149 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.027007 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17cab1-0794-408d-9d68-bd68708e3d39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.028290 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.028258 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.028408 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.028377 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ce4a34-1291-40bc-b3de-95ad93828326-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.028529 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.028507 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.028616 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.028597 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.081688 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.081650 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.082051 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.082031 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bn5g\" (UniqueName: \"kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-kube-api-access-8bn5g\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.082340 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.082322 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smdh6\" (UniqueName: \"kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-kube-api-access-smdh6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bzd25\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.082653 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.082630 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.129065 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.129034 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:46.381911 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.381880 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:46.554107 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.554050 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr"] Nov 03 01:03:46.557022 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:03:46.556993 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c17cab1_0794_408d_9d68_bd68708e3d39.slice/crio-7f4a575c6f2b3cfa7be15acd5df667d8059825b1d465e184ba5519a23ba3db60 WatchSource:0}: Error finding container 7f4a575c6f2b3cfa7be15acd5df667d8059825b1d465e184ba5519a23ba3db60: Status 404 returned error can't find the container with id 7f4a575c6f2b3cfa7be15acd5df667d8059825b1d465e184ba5519a23ba3db60 Nov 03 01:03:46.559573 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.559533 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:03:46.559696 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.559609 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:03:46.559696 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.559641 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:03:46.760964 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.760940 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25"] Nov 03 01:03:46.762294 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:03:46.762270 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ce4a34_1291_40bc_b3de_95ad93828326.slice/crio-88df3c1a72dd484fc0b9cee727e1b085e387c4ec6fe15c7588e8e3d6ad359700 WatchSource:0}: Error finding container 88df3c1a72dd484fc0b9cee727e1b085e387c4ec6fe15c7588e8e3d6ad359700: Status 404 returned error can't find the container with id 88df3c1a72dd484fc0b9cee727e1b085e387c4ec6fe15c7588e8e3d6ad359700 Nov 03 01:03:46.765104 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.765066 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:03:46.765507 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.765139 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:03:46.765507 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.765175 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:03:46.879974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.879932 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:46.890802 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.889948 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" event={"ID":"6c17cab1-0794-408d-9d68-bd68708e3d39","Type":"ContainerStarted","Data":"b082280d6ca603c7478241b3d153aff3c2bbe3321943b932bbd6343e949e2313"} Nov 03 01:03:46.890802 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.890016 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" event={"ID":"6c17cab1-0794-408d-9d68-bd68708e3d39","Type":"ContainerStarted","Data":"7f4a575c6f2b3cfa7be15acd5df667d8059825b1d465e184ba5519a23ba3db60"} Nov 03 01:03:46.892300 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.892260 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" event={"ID":"26ce4a34-1291-40bc-b3de-95ad93828326","Type":"ContainerStarted","Data":"982c7e7e1820757336f60c78bc0361a5d0deca679cf3d7705e05c19a5969126e"} Nov 03 01:03:46.892433 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:46.892318 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" event={"ID":"26ce4a34-1291-40bc-b3de-95ad93828326","Type":"ContainerStarted","Data":"88df3c1a72dd484fc0b9cee727e1b085e387c4ec6fe15c7588e8e3d6ad359700"} Nov 03 01:03:47.077018 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:47.076985 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:47.128345 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:47.128280 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" podStartSLOduration=2.128258226 podStartE2EDuration="2.128258226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:47.011008534 +0000 UTC m=+3473.961948695" watchObservedRunningTime="2026-11-03 01:03:47.128258226 +0000 UTC m=+3474.079198386" Nov 03 01:03:47.129814 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:47.129760 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" podStartSLOduration=2.12974413 podStartE2EDuration="2.12974413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:47.126241827 +0000 UTC m=+3474.077181998" watchObservedRunningTime="2026-11-03 01:03:47.12974413 +0000 UTC m=+3474.080684290" Nov 03 01:03:47.130671 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:47.130649 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:47.132389 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:47.132359 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 01:03:47.132542 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:47.132413 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 01:03:47.248085 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:47.248045 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:47.386330 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:47.386164 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:47.396305 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:47.395957 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 01:03:47.396305 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:47.396019 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 01:03:48.131869 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:48.131598 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 01:03:48.131869 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:48.131667 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 01:03:48.391093 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:48.385245 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 01:03:48.391093 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:48.385310 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 01:03:49.130223 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:49.130173 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 01:03:49.130451 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:49.130254 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 01:03:49.396913 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:49.389238 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 01:03:49.396913 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:49.389301 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 01:03:50.130326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:50.130180 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 01:03:50.130326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:50.130247 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 01:03:50.384898 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:50.384166 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 01:03:50.384898 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:50.384222 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 01:03:51.130849 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:51.129653 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 01:03:51.130849 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:51.129715 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 01:03:51.397900 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:51.397216 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 01:03:51.397900 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:51.397289 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 01:03:52.129636 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:52.129313 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 01:03:52.129636 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:52.129382 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 01:03:52.393900 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:52.385246 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 01:03:52.393900 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:52.385330 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 01:03:53.015886 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:53.015841 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:53.129404 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:53.129371 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 01:03:53.129584 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:53.129417 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 01:03:53.382656 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:53.382621 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 01:03:53.382870 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:53.382685 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 01:03:54.129708 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:54.129675 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 03 01:03:54.130064 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:54.129722 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 03 01:03:54.382881 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:54.382791 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 01:03:54.382881 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:54.382852 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 01:03:55.711512 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:55.711480 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:55.712247 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:55.712222 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:55.713582 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:55.713557 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:03:55.881720 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:55.881672 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:55.882734 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:55.882694 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:55.883661 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:55.883641 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:03:57.151925 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:03:57.151884 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:03:57.906265 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:03:57.906233 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:02.626472 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:02.626434 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:07.394229 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:07.394190 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:09.976952 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:09.976915 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:11.357895 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.357848 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:11.851203 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.851161 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd"] Nov 03 01:04:11.856905 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.856877 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:11.961547 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.961505 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd"] Nov 03 01:04:11.962191 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.962169 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4bfaef-openshift-default-dockercfg-2z8rz\"" Nov 03 01:04:11.974578 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.963053 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-token\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:11.974975 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.974944 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-data\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:11.981931 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.975190 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-podinfo\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:11.982437 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.982406 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-socket\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:11.982664 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.982646 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-certs\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:11.982794 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.982778 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-credential-socket\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:11.982985 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.982959 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzghh\" (UniqueName: \"kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-kube-api-access-tzghh\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:11.983117 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.983103 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-envoy\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:11.983278 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.983256 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098e3787-f743-4111-b1d0-9f8806ce3c48-istiod-ca-cert\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:11.991998 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:11.991970 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:12.119882 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.096532 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-data\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.121194 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.121163 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-podinfo\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.121701 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.121677 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-socket\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.122077 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.122059 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-certs\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.126167 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.126110 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-credential-socket\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.134053 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.134005 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzghh\" (UniqueName: \"kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-kube-api-access-tzghh\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.165881 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.148487 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-envoy\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.165881 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.148648 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098e3787-f743-4111-b1d0-9f8806ce3c48-istiod-ca-cert\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.165881 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.148738 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-token\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.165881 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.152055 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098e3787-f743-4111-b1d0-9f8806ce3c48-istiod-ca-cert\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.165881 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.157715 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-podinfo\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.166901 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.166513 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-data\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.166901 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.166565 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-certs\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.166901 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.166842 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-credential-socket\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.167408 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.167384 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-socket\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.170840 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.169428 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-envoy\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.191891 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.191813 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzghh\" (UniqueName: \"kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-kube-api-access-tzghh\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.207862 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.207811 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-token\") pod \"e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.490113 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.490002 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:12.549439 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.549336 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:12.698530 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.698359 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:12.981207 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.975153 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd"] Nov 03 01:04:12.993348 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:04:12.984950 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098e3787_f743_4111_b1d0_9f8806ce3c48.slice/crio-bc0855468b31a86f1af46b9db5f12b10db71d9ef6f1a6f06cbd69ee7de3c3ef1 WatchSource:0}: Error finding container bc0855468b31a86f1af46b9db5f12b10db71d9ef6f1a6f06cbd69ee7de3c3ef1: Status 404 returned error can't find the container with id bc0855468b31a86f1af46b9db5f12b10db71d9ef6f1a6f06cbd69ee7de3c3ef1 Nov 03 01:04:12.995782 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.995728 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:04:12.995923 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.995843 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:04:12.995923 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:12.995885 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:04:13.164275 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:13.164232 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" event={"ID":"098e3787-f743-4111-b1d0-9f8806ce3c48","Type":"ContainerStarted","Data":"a6ce21300cea96924783beb60e826194a3644848b087c6c9acc3a32069cc09c1"} Nov 03 01:04:13.164477 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:13.164307 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" event={"ID":"098e3787-f743-4111-b1d0-9f8806ce3c48","Type":"ContainerStarted","Data":"bc0855468b31a86f1af46b9db5f12b10db71d9ef6f1a6f06cbd69ee7de3c3ef1"} Nov 03 01:04:13.495849 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:13.494966 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:14.509846 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:14.507151 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:14.727928 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:14.727856 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" podStartSLOduration=3.727815152 podStartE2EDuration="3.727815152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:13.288616563 +0000 UTC m=+3500.239556720" watchObservedRunningTime="2026-11-03 01:04:14.727815152 +0000 UTC m=+3501.678755310" Nov 03 01:04:15.180094 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:15.180067 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:15.180280 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:15.180153 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:04:16.225874 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:16.225806 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:16.410606 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:16.410565 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:16.575198 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:16.575162 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:17.756161 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:17.756123 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:18.466655 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:18.466621 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:19.692523 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:19.692486 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:20.934733 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:20.934694 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:22.673665 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:22.673632 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:27.151738 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:27.151702 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:04:27.400115 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:27.400069 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:32.074681 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:32.074594 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:35.027957 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:35.027926 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:35.210865 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:35.202585 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:36.041889 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.041849 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn"] Nov 03 01:04:36.045884 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.045861 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.062114 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.062088 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:04:36.078709 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.078682 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp94m\" (UniqueName: \"kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-kube-api-access-fp94m\") pod \"payload-processing-external-model-84dd5cddcb-gmhsn\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.078858 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.078747 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-routing-overlay-volume\") pod \"payload-processing-external-model-84dd5cddcb-gmhsn\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.078858 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.078845 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials\") pod \"payload-processing-external-model-84dd5cddcb-gmhsn\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.078996 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.078978 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-plugins-config-volume\") pod \"payload-processing-external-model-84dd5cddcb-gmhsn\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.109278 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.109255 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e535ec6795f9\"" Nov 03 01:04:36.109451 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.109276 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2302a5a10819\"" Nov 03 01:04:36.109538 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.109319 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 01:04:36.109615 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.109553 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mxnb8\"" Nov 03 01:04:36.109671 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.109585 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 01:04:36.109742 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.109729 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 01:04:36.109848 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.109813 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 01:04:36.150465 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.150432 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn"] Nov 03 01:04:36.179962 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.179934 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-plugins-config-volume\") pod \"payload-processing-external-model-84dd5cddcb-gmhsn\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.180100 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.180015 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp94m\" (UniqueName: \"kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-kube-api-access-fp94m\") pod \"payload-processing-external-model-84dd5cddcb-gmhsn\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.180100 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.180076 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-routing-overlay-volume\") pod \"payload-processing-external-model-84dd5cddcb-gmhsn\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.180215 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.180151 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials\") pod \"payload-processing-external-model-84dd5cddcb-gmhsn\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.180884 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.180852 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-plugins-config-volume\") pod \"payload-processing-external-model-84dd5cddcb-gmhsn\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.181365 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.181339 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-routing-overlay-volume\") pod \"payload-processing-external-model-84dd5cddcb-gmhsn\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.183019 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.182996 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials\") pod \"payload-processing-external-model-84dd5cddcb-gmhsn\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.236738 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.236712 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp94m\" (UniqueName: \"kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-kube-api-access-fp94m\") pod \"payload-processing-external-model-84dd5cddcb-gmhsn\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.361604 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.361544 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:36.734074 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.734048 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn"] Nov 03 01:04:36.736165 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:04:36.736133 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5966f26_a784_494e_9e5c_0b0747134fce.slice/crio-9e3d34c38cd76aea8654d0a734802c5059b5b7983d6da965c4505d7567b95781 WatchSource:0}: Error finding container 9e3d34c38cd76aea8654d0a734802c5059b5b7983d6da965c4505d7567b95781: Status 404 returned error can't find the container with id 9e3d34c38cd76aea8654d0a734802c5059b5b7983d6da965c4505d7567b95781 Nov 03 01:04:36.816883 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:36.816855 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:37.254649 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:37.254618 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:37.323544 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:37.323508 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" event={"ID":"f5966f26-a784-494e-9e5c-0b0747134fce","Type":"ContainerStarted","Data":"02e4efdc2b20c6c26cf1a9a15de749d8626770879eb13c1f4d0e58d165f1456b"} Nov 03 01:04:37.323706 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:37.323576 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" event={"ID":"f5966f26-a784-494e-9e5c-0b0747134fce","Type":"ContainerStarted","Data":"9e3d34c38cd76aea8654d0a734802c5059b5b7983d6da965c4505d7567b95781"} Nov 03 01:04:37.323706 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:37.323603 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:37.437592 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:37.437541 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" podStartSLOduration=2.437528462 podStartE2EDuration="2.437528462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:37.435882536 +0000 UTC m=+3524.386822695" watchObservedRunningTime="2026-11-03 01:04:37.437528462 +0000 UTC m=+3524.388468618" Nov 03 01:04:37.855807 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:37.855763 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:38.031644 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:38.031610 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:38.188309 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:38.188284 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:38.443989 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:38.443902 6400 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:38.443989 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:38.443928 6400 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn: secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:38.443989 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:38.443992 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials podName:f5966f26-a784-494e-9e5c-0b0747134fce nodeName:}" failed. No retries permitted until 2026-11-03 01:04:38.943975508 +0000 UTC m=+3525.894915659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials") pod "payload-processing-external-model-84dd5cddcb-gmhsn" (UID: "f5966f26-a784-494e-9e5c-0b0747134fce") : secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:38.949266 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:38.949228 6400 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:38.949266 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:38.949258 6400 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn: secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:38.949536 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:38.949308 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials podName:f5966f26-a784-494e-9e5c-0b0747134fce nodeName:}" failed. No retries permitted until 2026-11-03 01:04:39.949294701 +0000 UTC m=+3526.900234836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials") pod "payload-processing-external-model-84dd5cddcb-gmhsn" (UID: "f5966f26-a784-494e-9e5c-0b0747134fce") : secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:39.961105 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:39.961068 6400 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:39.961105 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:39.961100 6400 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn: secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:39.961515 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:39.961162 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials podName:f5966f26-a784-494e-9e5c-0b0747134fce nodeName:}" failed. No retries permitted until 2026-11-03 01:04:41.961146757 +0000 UTC m=+3528.912086892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials") pod "payload-processing-external-model-84dd5cddcb-gmhsn" (UID: "f5966f26-a784-494e-9e5c-0b0747134fce") : secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:40.119945 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:40.119907 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:41.093116 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:41.093070 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:41.283678 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:41.283633 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn"] Nov 03 01:04:41.285972 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:41.285940 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:41.287656 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:41.287622 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" podUID="f5966f26-a784-494e-9e5c-0b0747134fce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:04:41.902308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:41.902274 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:41.985094 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:41.985067 6400 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:41.985094 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:41.985090 6400 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn: secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:41.985302 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:41.985140 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials podName:f5966f26-a784-494e-9e5c-0b0747134fce nodeName:}" failed. No retries permitted until 2026-11-03 01:04:45.985125298 +0000 UTC m=+3532.936065433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials") pod "payload-processing-external-model-84dd5cddcb-gmhsn" (UID: "f5966f26-a784-494e-9e5c-0b0747134fce") : secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:42.176276 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:42.176206 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:42.351749 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:42.351720 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:42.524182 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:42.524095 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:45.438378 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:45.438339 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:45.606099 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:45.606068 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:45.968751 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:45.968704 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg"] Nov 03 01:04:45.979785 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:45.976763 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" Nov 03 01:04:46.024716 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.024684 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg"] Nov 03 01:04:46.047587 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:46.047558 6400 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:46.047587 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:46.047585 6400 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn: secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:46.047889 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:46.047658 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials podName:f5966f26-a784-494e-9e5c-0b0747134fce nodeName:}" failed. No retries permitted until 2026-11-03 01:04:54.04763931 +0000 UTC m=+3540.998579460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials") pod "payload-processing-external-model-84dd5cddcb-gmhsn" (UID: "f5966f26-a784-494e-9e5c-0b0747134fce") : secret "agc-cred-e535ec6795f9" not found Nov 03 01:04:46.063478 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.063438 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-67b917\"" Nov 03 01:04:46.063658 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.063496 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-67b917-dockercfg-x4ggq\"" Nov 03 01:04:46.069764 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.069725 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:46.148736 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.148704 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzq8\" (UniqueName: \"kubernetes.io/projected/00dbe25c-0c20-424f-945c-f8e0dfd93f49-kube-api-access-gbzq8\") pod \"payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg\" (UID: \"00dbe25c-0c20-424f-945c-f8e0dfd93f49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" Nov 03 01:04:46.148953 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.148925 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00dbe25c-0c20-424f-945c-f8e0dfd93f49-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg\" (UID: \"00dbe25c-0c20-424f-945c-f8e0dfd93f49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" Nov 03 01:04:46.250388 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.250299 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00dbe25c-0c20-424f-945c-f8e0dfd93f49-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg\" (UID: \"00dbe25c-0c20-424f-945c-f8e0dfd93f49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" Nov 03 01:04:46.250388 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.250384 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzq8\" (UniqueName: \"kubernetes.io/projected/00dbe25c-0c20-424f-945c-f8e0dfd93f49-kube-api-access-gbzq8\") pod \"payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg\" (UID: \"00dbe25c-0c20-424f-945c-f8e0dfd93f49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" Nov 03 01:04:46.251055 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.251033 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00dbe25c-0c20-424f-945c-f8e0dfd93f49-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg\" (UID: \"00dbe25c-0c20-424f-945c-f8e0dfd93f49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" Nov 03 01:04:46.275201 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.275171 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:46.284335 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.284298 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" podUID="f5966f26-a784-494e-9e5c-0b0747134fce" containerName="payload-processing" containerID="cri-o://02e4efdc2b20c6c26cf1a9a15de749d8626770879eb13c1f4d0e58d165f1456b" gracePeriod=25 Nov 03 01:04:46.312445 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.312412 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzq8\" (UniqueName: \"kubernetes.io/projected/00dbe25c-0c20-424f-945c-f8e0dfd93f49-kube-api-access-gbzq8\") pod \"payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg\" (UID: \"00dbe25c-0c20-424f-945c-f8e0dfd93f49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" Nov 03 01:04:46.329357 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.329330 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" Nov 03 01:04:46.505782 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.505699 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:46.570905 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.570881 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:46.714024 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:04:46.713989 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00dbe25c_0c20_424f_945c_f8e0dfd93f49.slice/crio-a724510801dcbc053c245ee616edc95a3ec981219857160e0241d6d718c024a4 WatchSource:0}: Error finding container a724510801dcbc053c245ee616edc95a3ec981219857160e0241d6d718c024a4: Status 404 returned error can't find the container with id a724510801dcbc053c245ee616edc95a3ec981219857160e0241d6d718c024a4 Nov 03 01:04:46.737968 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.737941 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg"] Nov 03 01:04:46.757913 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.757849 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-routing-overlay-volume\") pod \"f5966f26-a784-494e-9e5c-0b0747134fce\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " Nov 03 01:04:46.758039 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.757929 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials\") pod \"f5966f26-a784-494e-9e5c-0b0747134fce\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " Nov 03 01:04:46.758101 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.758072 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-plugins-config-volume\") pod \"f5966f26-a784-494e-9e5c-0b0747134fce\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " Nov 03 01:04:46.758198 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.758176 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-routing-overlay-volume" pod "f5966f26-a784-494e-9e5c-0b0747134fce" (UID: "f5966f26-a784-494e-9e5c-0b0747134fce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:46.758257 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.758230 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-kube-api-access-fp94m\" (UniqueName: \"kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-kube-api-access-fp94m\") pod \"f5966f26-a784-494e-9e5c-0b0747134fce\" (UID: \"f5966f26-a784-494e-9e5c-0b0747134fce\") " Nov 03 01:04:46.758497 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.758473 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-plugins-config-volume" pod "f5966f26-a784-494e-9e5c-0b0747134fce" (UID: "f5966f26-a784-494e-9e5c-0b0747134fce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:46.758649 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.758631 6400 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-routing-overlay-volume\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:04:46.758716 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.758684 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5966f26-a784-494e-9e5c-0b0747134fce-plugins-config-volume\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:04:46.760415 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.760394 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials" pod "f5966f26-a784-494e-9e5c-0b0747134fce" (UID: "f5966f26-a784-494e-9e5c-0b0747134fce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:46.760415 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.760404 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-kube-api-access-fp94m" pod "f5966f26-a784-494e-9e5c-0b0747134fce" (UID: "f5966f26-a784-494e-9e5c-0b0747134fce"). InnerVolumeSpecName "kube-api-access-fp94m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:46.859975 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.859943 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp94m\" (UniqueName: \"kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-kube-api-access-fp94m\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:04:46.859975 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:46.859969 6400 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5966f26-a784-494e-9e5c-0b0747134fce-provider-credentials\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:04:47.213782 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.213752 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:47.383422 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.383388 6400 generic.go:356] "Generic (PLEG): container finished" podID="f5966f26-a784-494e-9e5c-0b0747134fce" containerID="02e4efdc2b20c6c26cf1a9a15de749d8626770879eb13c1f4d0e58d165f1456b" exitCode=0 Nov 03 01:04:47.383598 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.383443 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" Nov 03 01:04:47.383598 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.383465 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" event={"ID":"f5966f26-a784-494e-9e5c-0b0747134fce","Type":"ContainerDied","Data":"02e4efdc2b20c6c26cf1a9a15de749d8626770879eb13c1f4d0e58d165f1456b"} Nov 03 01:04:47.383598 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.383512 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn" event={"ID":"f5966f26-a784-494e-9e5c-0b0747134fce","Type":"ContainerDied","Data":"9e3d34c38cd76aea8654d0a734802c5059b5b7983d6da965c4505d7567b95781"} Nov 03 01:04:47.383598 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.383540 6400 scope.go:122] "RemoveContainer" containerID="02e4efdc2b20c6c26cf1a9a15de749d8626770879eb13c1f4d0e58d165f1456b" Nov 03 01:04:47.385244 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.385220 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" event={"ID":"00dbe25c-0c20-424f-945c-f8e0dfd93f49","Type":"ContainerStarted","Data":"d2859a4513f65e3d8eb27798fd05fe12651d284dcf4331b6812931b6ebc6cdbc"} Nov 03 01:04:47.385382 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.385262 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" event={"ID":"00dbe25c-0c20-424f-945c-f8e0dfd93f49","Type":"ContainerStarted","Data":"a724510801dcbc053c245ee616edc95a3ec981219857160e0241d6d718c024a4"} Nov 03 01:04:47.385382 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.385324 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" Nov 03 01:04:47.392719 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.392701 6400 scope.go:122] "RemoveContainer" containerID="02e4efdc2b20c6c26cf1a9a15de749d8626770879eb13c1f4d0e58d165f1456b" Nov 03 01:04:47.393019 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:47.392994 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e4efdc2b20c6c26cf1a9a15de749d8626770879eb13c1f4d0e58d165f1456b\": container with ID starting with 02e4efdc2b20c6c26cf1a9a15de749d8626770879eb13c1f4d0e58d165f1456b not found: ID does not exist" containerID="02e4efdc2b20c6c26cf1a9a15de749d8626770879eb13c1f4d0e58d165f1456b" Nov 03 01:04:47.393122 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.393028 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e4efdc2b20c6c26cf1a9a15de749d8626770879eb13c1f4d0e58d165f1456b"} err="failed to get container status \"02e4efdc2b20c6c26cf1a9a15de749d8626770879eb13c1f4d0e58d165f1456b\": rpc error: code = NotFound desc = could not find container \"02e4efdc2b20c6c26cf1a9a15de749d8626770879eb13c1f4d0e58d165f1456b\": container with ID starting with 02e4efdc2b20c6c26cf1a9a15de749d8626770879eb13c1f4d0e58d165f1456b not found: ID does not exist" Nov 03 01:04:47.547903 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.547865 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn"] Nov 03 01:04:47.610843 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.606504 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dd5cddcb-gmhsn"] Nov 03 01:04:47.687836 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:47.687768 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" podStartSLOduration=2.687754612 podStartE2EDuration="2.687754612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:47.686533258 +0000 UTC m=+3534.637473416" watchObservedRunningTime="2026-11-03 01:04:47.687754612 +0000 UTC m=+3534.638694791" Nov 03 01:04:49.178682 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:49.178645 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5966f26-a784-494e-9e5c-0b0747134fce" path="/var/lib/kubelet/pods/f5966f26-a784-494e-9e5c-0b0747134fce/volumes" Nov 03 01:04:50.325638 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.325587 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:50.589837 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.589744 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:50.760073 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.760034 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns"] Nov 03 01:04:50.761350 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.761323 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f5966f26-a784-494e-9e5c-0b0747134fce" containerName="payload-processing" Nov 03 01:04:50.761540 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.761525 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5966f26-a784-494e-9e5c-0b0747134fce" containerName="payload-processing" Nov 03 01:04:50.761941 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.761924 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5966f26-a784-494e-9e5c-0b0747134fce" containerName="payload-processing" Nov 03 01:04:50.767922 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.767903 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" Nov 03 01:04:50.804799 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.804757 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp54t\" (UniqueName: \"kubernetes.io/projected/314a73d2-e55f-4e16-afc1-3a312f0c41cc-kube-api-access-rp54t\") pod \"payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns\" (UID: \"314a73d2-e55f-4e16-afc1-3a312f0c41cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" Nov 03 01:04:50.805722 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.804991 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314a73d2-e55f-4e16-afc1-3a312f0c41cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns\" (UID: \"314a73d2-e55f-4e16-afc1-3a312f0c41cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" Nov 03 01:04:50.821592 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.821551 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns"] Nov 03 01:04:50.904904 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.903404 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-244924-dockercfg-2cxw5\"" Nov 03 01:04:50.904904 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.903507 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-244924\"" Nov 03 01:04:50.914849 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.908878 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp54t\" (UniqueName: \"kubernetes.io/projected/314a73d2-e55f-4e16-afc1-3a312f0c41cc-kube-api-access-rp54t\") pod \"payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns\" (UID: \"314a73d2-e55f-4e16-afc1-3a312f0c41cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" Nov 03 01:04:50.914849 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.909002 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314a73d2-e55f-4e16-afc1-3a312f0c41cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns\" (UID: \"314a73d2-e55f-4e16-afc1-3a312f0c41cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" Nov 03 01:04:50.914849 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.910006 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314a73d2-e55f-4e16-afc1-3a312f0c41cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns\" (UID: \"314a73d2-e55f-4e16-afc1-3a312f0c41cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" Nov 03 01:04:50.925572 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.916612 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph"] Nov 03 01:04:50.927870 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.926009 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" Nov 03 01:04:50.944340 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.944310 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph"] Nov 03 01:04:50.968908 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:50.968877 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp54t\" (UniqueName: \"kubernetes.io/projected/314a73d2-e55f-4e16-afc1-3a312f0c41cc-kube-api-access-rp54t\") pod \"payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns\" (UID: \"314a73d2-e55f-4e16-afc1-3a312f0c41cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" Nov 03 01:04:51.009779 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:51.009737 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zbtl\" (UniqueName: \"kubernetes.io/projected/db3defa8-9b42-4487-a9b4-070cf3863f9e-kube-api-access-7zbtl\") pod \"payload-processing-e2e-inf-a-244924-585fcf687-j8jph\" (UID: \"db3defa8-9b42-4487-a9b4-070cf3863f9e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" Nov 03 01:04:51.009996 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:51.009974 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3defa8-9b42-4487-a9b4-070cf3863f9e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-244924-585fcf687-j8jph\" (UID: \"db3defa8-9b42-4487-a9b4-070cf3863f9e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" Nov 03 01:04:51.090894 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:51.090848 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" Nov 03 01:04:51.111562 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:51.111534 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3defa8-9b42-4487-a9b4-070cf3863f9e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-244924-585fcf687-j8jph\" (UID: \"db3defa8-9b42-4487-a9b4-070cf3863f9e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" Nov 03 01:04:51.111681 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:51.111637 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zbtl\" (UniqueName: \"kubernetes.io/projected/db3defa8-9b42-4487-a9b4-070cf3863f9e-kube-api-access-7zbtl\") pod \"payload-processing-e2e-inf-a-244924-585fcf687-j8jph\" (UID: \"db3defa8-9b42-4487-a9b4-070cf3863f9e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" Nov 03 01:04:51.112269 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:51.112248 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3defa8-9b42-4487-a9b4-070cf3863f9e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-244924-585fcf687-j8jph\" (UID: \"db3defa8-9b42-4487-a9b4-070cf3863f9e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" Nov 03 01:04:51.174664 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:51.174637 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zbtl\" (UniqueName: \"kubernetes.io/projected/db3defa8-9b42-4487-a9b4-070cf3863f9e-kube-api-access-7zbtl\") pod \"payload-processing-e2e-inf-a-244924-585fcf687-j8jph\" (UID: \"db3defa8-9b42-4487-a9b4-070cf3863f9e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" Nov 03 01:04:51.250244 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:51.250214 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" Nov 03 01:04:51.479323 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:51.479300 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns"] Nov 03 01:04:51.480850 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:04:51.480795 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314a73d2_e55f_4e16_afc1_3a312f0c41cc.slice/crio-662ffeef79b91cf3223eea87d2472d1b18778c41ee220d305a078723ffbb25a2 WatchSource:0}: Error finding container 662ffeef79b91cf3223eea87d2472d1b18778c41ee220d305a078723ffbb25a2: Status 404 returned error can't find the container with id 662ffeef79b91cf3223eea87d2472d1b18778c41ee220d305a078723ffbb25a2 Nov 03 01:04:51.624119 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:51.624094 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph"] Nov 03 01:04:51.626155 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:04:51.626130 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3defa8_9b42_4487_a9b4_070cf3863f9e.slice/crio-e816c63971f6b0fdf1dc8f3fe53c499a21ab0d412a2ca8ac4911148e5745c15f WatchSource:0}: Error finding container e816c63971f6b0fdf1dc8f3fe53c499a21ab0d412a2ca8ac4911148e5745c15f: Status 404 returned error can't find the container with id e816c63971f6b0fdf1dc8f3fe53c499a21ab0d412a2ca8ac4911148e5745c15f Nov 03 01:04:52.010006 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:52.009972 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:52.463681 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:52.463643 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" event={"ID":"314a73d2-e55f-4e16-afc1-3a312f0c41cc","Type":"ContainerStarted","Data":"b6776db70a694314ada71edb33b031fce2f03f9b95baf923c0d54e791fd0d2fb"} Nov 03 01:04:52.463938 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:52.463704 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" event={"ID":"314a73d2-e55f-4e16-afc1-3a312f0c41cc","Type":"ContainerStarted","Data":"662ffeef79b91cf3223eea87d2472d1b18778c41ee220d305a078723ffbb25a2"} Nov 03 01:04:52.463938 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:52.463805 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" Nov 03 01:04:52.465244 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:52.465212 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" event={"ID":"db3defa8-9b42-4487-a9b4-070cf3863f9e","Type":"ContainerStarted","Data":"2a770f3f669833d4146ae25433caaaa6303060db0a9d5a29643c3e5596a38926"} Nov 03 01:04:52.465363 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:52.465251 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" event={"ID":"db3defa8-9b42-4487-a9b4-070cf3863f9e","Type":"ContainerStarted","Data":"e816c63971f6b0fdf1dc8f3fe53c499a21ab0d412a2ca8ac4911148e5745c15f"} Nov 03 01:04:52.465363 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:52.465353 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" Nov 03 01:04:52.577181 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:52.577124 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" podStartSLOduration=2.577109116 podStartE2EDuration="2.577109116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:52.57575663 +0000 UTC m=+3539.526696787" watchObservedRunningTime="2026-11-03 01:04:52.577109116 +0000 UTC m=+3539.528049274" Nov 03 01:04:52.705887 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:52.703263 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" podStartSLOduration=2.703238938 podStartE2EDuration="2.703238938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:52.697227283 +0000 UTC m=+3539.648167441" watchObservedRunningTime="2026-11-03 01:04:52.703238938 +0000 UTC m=+3539.654179093" Nov 03 01:04:56.630850 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:56.626230 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:56.796060 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:56.796023 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:56.951410 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:56.951327 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:57.151479 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:04:57.151442 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:04:58.393986 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:04:58.393959 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" Nov 03 01:05:01.664811 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:01.664752 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:02.986888 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:02.986852 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:03.161349 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:03.161306 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:03.473769 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:03.473743 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" Nov 03 01:05:03.473977 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:03.473810 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" Nov 03 01:05:06.120746 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:06.120701 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:06.402697 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:06.402611 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:11.105677 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:11.105636 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:15.830187 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:15.830148 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:18.617725 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:18.617691 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:18.801579 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:18.801529 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:20.267916 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:20.267886 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:05:20.475442 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:20.475412 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:25.134357 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:25.134299 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:27.151512 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:05:27.151478 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:05:29.787317 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:29.787282 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:34.225589 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.225559 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7"] Nov 03 01:05:34.230159 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.230140 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:34.332447 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.332418 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-4bfaef-serving-cert\"" Nov 03 01:05:34.334810 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.334789 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7"] Nov 03 01:05:34.391637 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.391609 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96dr8\" (UniqueName: \"kubernetes.io/projected/98f1b4dc-d5fb-402b-a75d-37a558b05167-kube-api-access-96dr8\") pod \"maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:34.391771 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.391650 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-metrics-certs\") pod \"maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:34.391771 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.391699 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:34.391948 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.391906 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98f1b4dc-d5fb-402b-a75d-37a558b05167-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:34.446584 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.446550 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:34.501600 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.501519 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98f1b4dc-d5fb-402b-a75d-37a558b05167-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:34.501776 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.501680 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96dr8\" (UniqueName: \"kubernetes.io/projected/98f1b4dc-d5fb-402b-a75d-37a558b05167-kube-api-access-96dr8\") pod \"maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:34.501776 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.501731 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-metrics-certs\") pod \"maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:34.501939 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.501780 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:34.504288 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:05:34.502511 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-serving-cert: secret "maas-api-e2e-extmod-b-4bfaef-serving-cert" not found Nov 03 01:05:34.504288 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:05:34.502626 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-maas-api-tls podName:98f1b4dc-d5fb-402b-a75d-37a558b05167 nodeName:}" failed. No retries permitted until 2026-11-03 01:05:35.002606573 +0000 UTC m=+3581.953546722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-maas-api-tls") pod "maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" (UID: "98f1b4dc-d5fb-402b-a75d-37a558b05167") : secret "maas-api-e2e-extmod-b-4bfaef-serving-cert" not found Nov 03 01:05:34.504288 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.502877 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98f1b4dc-d5fb-402b-a75d-37a558b05167-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:34.508841 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.508184 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-metrics-certs\") pod \"maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:34.559602 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.559574 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96dr8\" (UniqueName: \"kubernetes.io/projected/98f1b4dc-d5fb-402b-a75d-37a558b05167-kube-api-access-96dr8\") pod \"maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:34.722479 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.722439 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:34.878252 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:34.878183 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:35.007419 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:35.007380 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:35.009975 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:35.009950 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:35.143635 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:35.143553 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:35.517235 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:35.517210 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7"] Nov 03 01:05:35.519071 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:05:35.519041 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f1b4dc_d5fb_402b_a75d_37a558b05167.slice/crio-ada5247e44ab91a8d95b30976801853667a961043ea44f83670662beb31ea3bc WatchSource:0}: Error finding container ada5247e44ab91a8d95b30976801853667a961043ea44f83670662beb31ea3bc: Status 404 returned error can't find the container with id ada5247e44ab91a8d95b30976801853667a961043ea44f83670662beb31ea3bc Nov 03 01:05:35.743847 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:35.743801 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" event={"ID":"98f1b4dc-d5fb-402b-a75d-37a558b05167","Type":"ContainerStarted","Data":"ada5247e44ab91a8d95b30976801853667a961043ea44f83670662beb31ea3bc"} Nov 03 01:05:36.751889 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:36.751797 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" event={"ID":"98f1b4dc-d5fb-402b-a75d-37a558b05167","Type":"ContainerStarted","Data":"8f5af126f70e4302f861954e6cadc335734fdd454d8851c73de65f8e3fe09136"} Nov 03 01:05:36.752300 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:36.752002 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:36.867040 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:36.866988 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" podStartSLOduration=2.036386407 podStartE2EDuration="2.866972069s" totalImagesPullingTime="830.585662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:36.86552483 +0000 UTC m=+3583.816464988" watchObservedRunningTime="2026-11-03 01:05:36.866972069 +0000 UTC m=+3583.817912224" Nov 03 01:05:39.449732 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:39.448951 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:42.762489 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:42.762460 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:05:44.211039 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:44.211000 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:48.905101 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:48.905062 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:50.203770 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:50.203738 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:50.363588 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:50.363550 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:53.579087 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:53.579053 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:56.507928 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:56.507898 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:05:57.152478 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:05:57.152438 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:05:58.301481 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:05:58.301435 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:02.734248 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.734212 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl"] Nov 03 01:06:02.739192 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.739166 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:02.758391 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.758359 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4a08b1-b2ad-4a97-8925-4e971934937a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:02.758547 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.758396 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmc2v\" (UniqueName: \"kubernetes.io/projected/1c4a08b1-b2ad-4a97-8925-4e971934937a-kube-api-access-gmc2v\") pod \"maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:02.758658 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.758633 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-metrics-certs\") pod \"maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:02.758766 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.758747 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-maas-api-tls\") pod \"maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:02.791232 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.791199 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-67b917-serving-cert\"" Nov 03 01:06:02.843364 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.843331 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl"] Nov 03 01:06:02.859610 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.859578 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4a08b1-b2ad-4a97-8925-4e971934937a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:02.859610 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.859614 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmc2v\" (UniqueName: \"kubernetes.io/projected/1c4a08b1-b2ad-4a97-8925-4e971934937a-kube-api-access-gmc2v\") pod \"maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:02.859881 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.859720 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-metrics-certs\") pod \"maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:02.859881 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.859759 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-maas-api-tls\") pod \"maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:02.859881 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:06:02.859877 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-serving-cert: secret "maas-api-e2e-inf-a-67b917-serving-cert" not found Nov 03 01:06:02.860045 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:06:02.859936 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-maas-api-tls podName:1c4a08b1-b2ad-4a97-8925-4e971934937a nodeName:}" failed. No retries permitted until 2026-11-03 01:06:03.359920195 +0000 UTC m=+3610.310860344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-maas-api-tls") pod "maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" (UID: "1c4a08b1-b2ad-4a97-8925-4e971934937a") : secret "maas-api-e2e-inf-a-67b917-serving-cert" not found Nov 03 01:06:02.860265 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.860238 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4a08b1-b2ad-4a97-8925-4e971934937a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:02.862363 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.862344 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-metrics-certs\") pod \"maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:02.917273 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.917242 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmc2v\" (UniqueName: \"kubernetes.io/projected/1c4a08b1-b2ad-4a97-8925-4e971934937a-kube-api-access-gmc2v\") pod \"maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:02.957624 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:02.957580 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:03.153811 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:03.153727 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:03.318598 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:03.318555 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:03.368216 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:03.368176 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-maas-api-tls\") pod \"maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:03.370768 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:03.370738 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-maas-api-tls\") pod \"maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:03.654968 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:03.654923 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:04.038337 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:04.038311 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl"] Nov 03 01:06:04.041085 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:06:04.041058 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4a08b1_b2ad_4a97_8925_4e971934937a.slice/crio-a8e49b1e814f73e034c767cf3dc66f9b121bc1d00e7b59cd163df0cc0b134ca5 WatchSource:0}: Error finding container a8e49b1e814f73e034c767cf3dc66f9b121bc1d00e7b59cd163df0cc0b134ca5: Status 404 returned error can't find the container with id a8e49b1e814f73e034c767cf3dc66f9b121bc1d00e7b59cd163df0cc0b134ca5 Nov 03 01:06:04.950978 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:04.950948 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" event={"ID":"1c4a08b1-b2ad-4a97-8925-4e971934937a","Type":"ContainerStarted","Data":"a8e49b1e814f73e034c767cf3dc66f9b121bc1d00e7b59cd163df0cc0b134ca5"} Nov 03 01:06:05.960246 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:05.960144 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" event={"ID":"1c4a08b1-b2ad-4a97-8925-4e971934937a","Type":"ContainerStarted","Data":"a5823d73090977dc1119cf72f6d5dfc208c9949aecd5ea1a81852a22cb7f68c1"} Nov 03 01:06:05.960688 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:05.960319 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:05.961726 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:05.961698 6400 generic.go:356] "Generic (PLEG): container finished" podID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerID="a6ce21300cea96924783beb60e826194a3644848b087c6c9acc3a32069cc09c1" exitCode=0 Nov 03 01:06:05.961833 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:05.961762 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" event={"ID":"098e3787-f743-4111-b1d0-9f8806ce3c48","Type":"ContainerDied","Data":"a6ce21300cea96924783beb60e826194a3644848b087c6c9acc3a32069cc09c1"} Nov 03 01:06:05.962135 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:05.962121 6400 scope.go:122] "RemoveContainer" containerID="a6ce21300cea96924783beb60e826194a3644848b087c6c9acc3a32069cc09c1" Nov 03 01:06:05.963645 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:05.963619 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:06:05.963702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:05.963686 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:06:05.963740 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:05.963718 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:06:06.075574 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:06.075508 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" podStartSLOduration=3.196580657 podStartE2EDuration="4.075489782s" totalImagesPullingTime="878.909125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:06.073990504 +0000 UTC m=+3613.024930673" watchObservedRunningTime="2026-11-03 01:06:06.075489782 +0000 UTC m=+3613.026429941" Nov 03 01:06:06.970154 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:06.970114 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" event={"ID":"098e3787-f743-4111-b1d0-9f8806ce3c48","Type":"ContainerStarted","Data":"c58b30ea2585fc5bfec2e71b5b3c46fbcb2606a1d08eb8ced61cf8638037845d"} Nov 03 01:06:07.496846 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:07.495885 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:06:07.496846 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:07.495917 6400 patch_prober.go:29] interesting pod/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 01:06:07.496846 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:07.495981 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 01:06:07.903845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:07.903473 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:08.501322 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:08.501239 6400 patch_prober.go:29] interesting pod/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 01:06:08.501322 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:08.501309 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 01:06:09.491273 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:09.491234 6400 patch_prober.go:29] interesting pod/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 01:06:09.491569 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:09.491539 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 01:06:10.496846 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:10.496224 6400 patch_prober.go:29] interesting pod/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 01:06:10.496846 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:10.496288 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 01:06:11.500979 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:11.500940 6400 patch_prober.go:29] interesting pod/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 01:06:11.501389 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:11.501000 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 01:06:11.976454 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:11.976423 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:06:12.495100 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:12.495073 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:06:12.495539 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:12.495523 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:06:12.496281 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:12.496260 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:06:12.537790 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:12.537762 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:12.764454 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:12.764343 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:12.920223 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:12.920171 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:17.561560 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:17.561522 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:19.146105 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:19.146064 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:22.309891 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:22.309810 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:27.028235 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:27.028199 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:27.152848 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:06:27.152781 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:06:31.798784 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:31.798751 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:32.088373 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:32.088279 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:32.246101 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:32.246060 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:32.646657 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:32.646612 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:33.069010 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:33.068974 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:33.723387 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:33.723224 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:34.028019 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:34.027932 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:36.906581 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:36.906550 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:39.045746 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:39.045711 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:39.274003 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:39.273975 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:42.218351 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:42.218307 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:42.886781 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:42.886746 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:43.104324 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:43.104292 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:46.935233 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:46.935192 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:49.837113 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:49.837069 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:06:51.700862 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:51.700799 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:56.365629 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:06:56.365593 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:57.154316 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:06:57.154272 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:07:01.066153 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:01.066115 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:05.811985 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:05.811951 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:10.653230 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:10.653186 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:15.364273 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:15.364233 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:15.678788 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:15.678708 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:19.500793 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:19.500753 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:20.465235 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:20.465200 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:20.875897 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:20.875851 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:21.081984 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:21.081955 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:21.724678 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:21.724634 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:25.503132 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:25.503097 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:27.154493 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:07:27.154449 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:07:27.421221 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:27.421140 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:30.217880 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:30.217786 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:34.921867 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:34.921773 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:39.711040 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:39.711009 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:41.343117 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:41.343074 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:44.175652 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:44.175622 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:44.552318 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:44.552289 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:47.267400 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:47.267354 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:47.947538 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:47.947503 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg"] Nov 03 01:07:48.341301 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:48.341258 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:48.514275 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:48.514217 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:49.728097 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:49.728061 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:49.955690 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:49.955659 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:50.412998 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:50.412962 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph"] Nov 03 01:07:50.520708 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:50.520655 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns"] Nov 03 01:07:50.816096 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:50.816062 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:50.985559 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:50.985522 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:52.789167 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:52.789127 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:52.948091 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:52.948029 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" podUID="00dbe25c-0c20-424f-945c-f8e0dfd93f49" containerName="payload-processing" containerID="cri-o://d2859a4513f65e3d8eb27798fd05fe12651d284dcf4331b6812931b6ebc6cdbc" gracePeriod=25 Nov 03 01:07:53.214116 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.214087 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" Nov 03 01:07:53.315783 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.315678 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00dbe25c-0c20-424f-945c-f8e0dfd93f49-kube-api-access-gbzq8\" (UniqueName: \"kubernetes.io/projected/00dbe25c-0c20-424f-945c-f8e0dfd93f49-kube-api-access-gbzq8\") pod \"00dbe25c-0c20-424f-945c-f8e0dfd93f49\" (UID: \"00dbe25c-0c20-424f-945c-f8e0dfd93f49\") " Nov 03 01:07:53.316007 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.315801 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00dbe25c-0c20-424f-945c-f8e0dfd93f49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00dbe25c-0c20-424f-945c-f8e0dfd93f49-plugins-config-volume\") pod \"00dbe25c-0c20-424f-945c-f8e0dfd93f49\" (UID: \"00dbe25c-0c20-424f-945c-f8e0dfd93f49\") " Nov 03 01:07:53.316182 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.316157 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00dbe25c-0c20-424f-945c-f8e0dfd93f49-plugins-config-volume" pod "00dbe25c-0c20-424f-945c-f8e0dfd93f49" (UID: "00dbe25c-0c20-424f-945c-f8e0dfd93f49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:07:53.316374 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.316357 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00dbe25c-0c20-424f-945c-f8e0dfd93f49-plugins-config-volume\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:07:53.318760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.318728 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00dbe25c-0c20-424f-945c-f8e0dfd93f49-kube-api-access-gbzq8" pod "00dbe25c-0c20-424f-945c-f8e0dfd93f49" (UID: "00dbe25c-0c20-424f-945c-f8e0dfd93f49"). InnerVolumeSpecName "kube-api-access-gbzq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:07:53.417195 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.417158 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzq8\" (UniqueName: \"kubernetes.io/projected/00dbe25c-0c20-424f-945c-f8e0dfd93f49-kube-api-access-gbzq8\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:07:53.675506 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.675414 6400 generic.go:356] "Generic (PLEG): container finished" podID="00dbe25c-0c20-424f-945c-f8e0dfd93f49" containerID="d2859a4513f65e3d8eb27798fd05fe12651d284dcf4331b6812931b6ebc6cdbc" exitCode=0 Nov 03 01:07:53.675506 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.675467 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" Nov 03 01:07:53.675701 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.675502 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" event={"ID":"00dbe25c-0c20-424f-945c-f8e0dfd93f49","Type":"ContainerDied","Data":"d2859a4513f65e3d8eb27798fd05fe12651d284dcf4331b6812931b6ebc6cdbc"} Nov 03 01:07:53.675701 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.675562 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg" event={"ID":"00dbe25c-0c20-424f-945c-f8e0dfd93f49","Type":"ContainerDied","Data":"a724510801dcbc053c245ee616edc95a3ec981219857160e0241d6d718c024a4"} Nov 03 01:07:53.675701 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.675596 6400 scope.go:122] "RemoveContainer" containerID="d2859a4513f65e3d8eb27798fd05fe12651d284dcf4331b6812931b6ebc6cdbc" Nov 03 01:07:53.686264 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.686246 6400 scope.go:122] "RemoveContainer" containerID="d2859a4513f65e3d8eb27798fd05fe12651d284dcf4331b6812931b6ebc6cdbc" Nov 03 01:07:53.686553 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:07:53.686533 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2859a4513f65e3d8eb27798fd05fe12651d284dcf4331b6812931b6ebc6cdbc\": container with ID starting with d2859a4513f65e3d8eb27798fd05fe12651d284dcf4331b6812931b6ebc6cdbc not found: ID does not exist" containerID="d2859a4513f65e3d8eb27798fd05fe12651d284dcf4331b6812931b6ebc6cdbc" Nov 03 01:07:53.686613 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.686563 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2859a4513f65e3d8eb27798fd05fe12651d284dcf4331b6812931b6ebc6cdbc"} err="failed to get container status \"d2859a4513f65e3d8eb27798fd05fe12651d284dcf4331b6812931b6ebc6cdbc\": rpc error: code = NotFound desc = could not find container \"d2859a4513f65e3d8eb27798fd05fe12651d284dcf4331b6812931b6ebc6cdbc\": container with ID starting with d2859a4513f65e3d8eb27798fd05fe12651d284dcf4331b6812931b6ebc6cdbc not found: ID does not exist" Nov 03 01:07:53.838715 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.838682 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg"] Nov 03 01:07:53.842175 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:53.842147 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-67b917-5c6c45475c-gssqg"] Nov 03 01:07:54.422178 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:54.422092 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:55.181101 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.181058 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00dbe25c-0c20-424f-945c-f8e0dfd93f49" path="/var/lib/kubelet/pods/00dbe25c-0c20-424f-945c-f8e0dfd93f49/volumes" Nov 03 01:07:55.413312 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.413269 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" podUID="db3defa8-9b42-4487-a9b4-070cf3863f9e" containerName="payload-processing" containerID="cri-o://2a770f3f669833d4146ae25433caaaa6303060db0a9d5a29643c3e5596a38926" gracePeriod=25 Nov 03 01:07:55.521467 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.521433 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" podUID="314a73d2-e55f-4e16-afc1-3a312f0c41cc" containerName="payload-pre-processing" containerID="cri-o://b6776db70a694314ada71edb33b031fce2f03f9b95baf923c0d54e791fd0d2fb" gracePeriod=25 Nov 03 01:07:55.525252 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.525217 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:55.691356 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.691278 6400 generic.go:356] "Generic (PLEG): container finished" podID="314a73d2-e55f-4e16-afc1-3a312f0c41cc" containerID="b6776db70a694314ada71edb33b031fce2f03f9b95baf923c0d54e791fd0d2fb" exitCode=0 Nov 03 01:07:55.691356 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.691345 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" event={"ID":"314a73d2-e55f-4e16-afc1-3a312f0c41cc","Type":"ContainerDied","Data":"b6776db70a694314ada71edb33b031fce2f03f9b95baf923c0d54e791fd0d2fb"} Nov 03 01:07:55.692706 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.692679 6400 generic.go:356] "Generic (PLEG): container finished" podID="db3defa8-9b42-4487-a9b4-070cf3863f9e" containerID="2a770f3f669833d4146ae25433caaaa6303060db0a9d5a29643c3e5596a38926" exitCode=0 Nov 03 01:07:55.692857 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.692749 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" event={"ID":"db3defa8-9b42-4487-a9b4-070cf3863f9e","Type":"ContainerDied","Data":"2a770f3f669833d4146ae25433caaaa6303060db0a9d5a29643c3e5596a38926"} Nov 03 01:07:55.692857 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.692788 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" event={"ID":"db3defa8-9b42-4487-a9b4-070cf3863f9e","Type":"ContainerDied","Data":"e816c63971f6b0fdf1dc8f3fe53c499a21ab0d412a2ca8ac4911148e5745c15f"} Nov 03 01:07:55.692857 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.692805 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e816c63971f6b0fdf1dc8f3fe53c499a21ab0d412a2ca8ac4911148e5745c15f" Nov 03 01:07:55.693534 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.693512 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" Nov 03 01:07:55.744907 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.744870 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db3defa8-9b42-4487-a9b4-070cf3863f9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3defa8-9b42-4487-a9b4-070cf3863f9e-plugins-config-volume\") pod \"db3defa8-9b42-4487-a9b4-070cf3863f9e\" (UID: \"db3defa8-9b42-4487-a9b4-070cf3863f9e\") " Nov 03 01:07:55.745229 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.745209 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3defa8-9b42-4487-a9b4-070cf3863f9e-kube-api-access-7zbtl\" (UniqueName: \"kubernetes.io/projected/db3defa8-9b42-4487-a9b4-070cf3863f9e-kube-api-access-7zbtl\") pod \"db3defa8-9b42-4487-a9b4-070cf3863f9e\" (UID: \"db3defa8-9b42-4487-a9b4-070cf3863f9e\") " Nov 03 01:07:55.745353 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.745325 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3defa8-9b42-4487-a9b4-070cf3863f9e-plugins-config-volume" pod "db3defa8-9b42-4487-a9b4-070cf3863f9e" (UID: "db3defa8-9b42-4487-a9b4-070cf3863f9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:07:55.745692 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.745670 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3defa8-9b42-4487-a9b4-070cf3863f9e-plugins-config-volume\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:07:55.747630 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.747603 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3defa8-9b42-4487-a9b4-070cf3863f9e-kube-api-access-7zbtl" pod "db3defa8-9b42-4487-a9b4-070cf3863f9e" (UID: "db3defa8-9b42-4487-a9b4-070cf3863f9e"). InnerVolumeSpecName "kube-api-access-7zbtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:07:55.807794 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.807766 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" Nov 03 01:07:55.846272 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.846236 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/314a73d2-e55f-4e16-afc1-3a312f0c41cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314a73d2-e55f-4e16-afc1-3a312f0c41cc-plugins-config-volume\") pod \"314a73d2-e55f-4e16-afc1-3a312f0c41cc\" (UID: \"314a73d2-e55f-4e16-afc1-3a312f0c41cc\") " Nov 03 01:07:55.846272 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.846274 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314a73d2-e55f-4e16-afc1-3a312f0c41cc-kube-api-access-rp54t\" (UniqueName: \"kubernetes.io/projected/314a73d2-e55f-4e16-afc1-3a312f0c41cc-kube-api-access-rp54t\") pod \"314a73d2-e55f-4e16-afc1-3a312f0c41cc\" (UID: \"314a73d2-e55f-4e16-afc1-3a312f0c41cc\") " Nov 03 01:07:55.846556 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.846539 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zbtl\" (UniqueName: \"kubernetes.io/projected/db3defa8-9b42-4487-a9b4-070cf3863f9e-kube-api-access-7zbtl\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:07:55.846691 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.846665 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314a73d2-e55f-4e16-afc1-3a312f0c41cc-plugins-config-volume" pod "314a73d2-e55f-4e16-afc1-3a312f0c41cc" (UID: "314a73d2-e55f-4e16-afc1-3a312f0c41cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:07:55.848643 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.848615 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314a73d2-e55f-4e16-afc1-3a312f0c41cc-kube-api-access-rp54t" pod "314a73d2-e55f-4e16-afc1-3a312f0c41cc" (UID: "314a73d2-e55f-4e16-afc1-3a312f0c41cc"). InnerVolumeSpecName "kube-api-access-rp54t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:07:55.947569 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.947486 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314a73d2-e55f-4e16-afc1-3a312f0c41cc-plugins-config-volume\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:07:55.947569 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:55.947516 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp54t\" (UniqueName: \"kubernetes.io/projected/314a73d2-e55f-4e16-afc1-3a312f0c41cc-kube-api-access-rp54t\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:07:56.703677 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:56.703601 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" Nov 03 01:07:56.703677 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:56.703636 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns" event={"ID":"314a73d2-e55f-4e16-afc1-3a312f0c41cc","Type":"ContainerDied","Data":"662ffeef79b91cf3223eea87d2472d1b18778c41ee220d305a078723ffbb25a2"} Nov 03 01:07:56.704172 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:56.703695 6400 scope.go:122] "RemoveContainer" containerID="b6776db70a694314ada71edb33b031fce2f03f9b95baf923c0d54e791fd0d2fb" Nov 03 01:07:56.704172 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:56.703860 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph" Nov 03 01:07:56.866093 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:56.866059 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns"] Nov 03 01:07:56.915342 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:56.915302 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-244924-5587ffc74c-j4dns"] Nov 03 01:07:56.977445 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:56.977364 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph"] Nov 03 01:07:56.982437 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:56.982413 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-244924-585fcf687-j8jph"] Nov 03 01:07:57.153100 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:07:57.153057 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:07:57.179475 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:57.179444 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314a73d2-e55f-4e16-afc1-3a312f0c41cc" path="/var/lib/kubelet/pods/314a73d2-e55f-4e16-afc1-3a312f0c41cc/volumes" Nov 03 01:07:57.179872 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:57.179858 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db3defa8-9b42-4487-a9b4-070cf3863f9e" path="/var/lib/kubelet/pods/db3defa8-9b42-4487-a9b4-070cf3863f9e/volumes" Nov 03 01:07:59.165485 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:07:59.164617 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:03.818514 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:03.818481 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:08.258226 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.258190 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh"] Nov 03 01:08:08.258714 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.258699 6400 cpu_manager.go:414] "containerMap: removing container" podUID="00dbe25c-0c20-424f-945c-f8e0dfd93f49" containerName="payload-processing" Nov 03 01:08:08.258714 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.258714 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00dbe25c-0c20-424f-945c-f8e0dfd93f49" containerName="payload-processing" Nov 03 01:08:08.258788 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.258726 6400 cpu_manager.go:414] "containerMap: removing container" podUID="314a73d2-e55f-4e16-afc1-3a312f0c41cc" containerName="payload-pre-processing" Nov 03 01:08:08.258788 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.258732 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314a73d2-e55f-4e16-afc1-3a312f0c41cc" containerName="payload-pre-processing" Nov 03 01:08:08.258788 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.258747 6400 cpu_manager.go:414] "containerMap: removing container" podUID="db3defa8-9b42-4487-a9b4-070cf3863f9e" containerName="payload-processing" Nov 03 01:08:08.258788 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.258752 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3defa8-9b42-4487-a9b4-070cf3863f9e" containerName="payload-processing" Nov 03 01:08:08.258953 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.258868 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="00dbe25c-0c20-424f-945c-f8e0dfd93f49" containerName="payload-processing" Nov 03 01:08:08.258953 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.258876 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="314a73d2-e55f-4e16-afc1-3a312f0c41cc" containerName="payload-pre-processing" Nov 03 01:08:08.258953 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.258883 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3defa8-9b42-4487-a9b4-070cf3863f9e" containerName="payload-processing" Nov 03 01:08:08.263534 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.263507 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.367118 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.367084 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b47e01-serving-cert\"" Nov 03 01:08:08.369384 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.369362 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh"] Nov 03 01:08:08.399399 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.399362 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-metrics-certs\") pod \"maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.399399 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.399399 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td5kj\" (UniqueName: \"kubernetes.io/projected/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-kube-api-access-td5kj\") pod \"maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.399671 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.399645 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.399711 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.399693 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-maas-api-tls\") pod \"maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.485619 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.485582 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:08.501301 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.501261 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.501524 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.501309 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-maas-api-tls\") pod \"maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.501524 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.501419 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-metrics-certs\") pod \"maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.501524 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.501447 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td5kj\" (UniqueName: \"kubernetes.io/projected/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-kube-api-access-td5kj\") pod \"maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.502167 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.502115 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.504207 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.504182 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-metrics-certs\") pod \"maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.504310 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.504230 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-maas-api-tls\") pod \"maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.563027 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.562991 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td5kj\" (UniqueName: \"kubernetes.io/projected/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-kube-api-access-td5kj\") pod \"maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.576631 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.576602 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:08.677002 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.676739 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:08.959058 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.959030 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh"] Nov 03 01:08:08.960776 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:08:08.960747 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35655276_9bb5_4f99_b6c9_c218c1e8b7d2.slice/crio-e6fce000f2dd09ed4c9d6fa739c5efbf2af074c41951935be688069b0a48815d WatchSource:0}: Error finding container e6fce000f2dd09ed4c9d6fa739c5efbf2af074c41951935be688069b0a48815d: Status 404 returned error can't find the container with id e6fce000f2dd09ed4c9d6fa739c5efbf2af074c41951935be688069b0a48815d Nov 03 01:08:08.962158 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:08.962143 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:08:09.779554 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:09.779513 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" event={"ID":"35655276-9bb5-4f99-b6c9-c218c1e8b7d2","Type":"ContainerStarted","Data":"e6fce000f2dd09ed4c9d6fa739c5efbf2af074c41951935be688069b0a48815d"} Nov 03 01:08:10.786923 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:10.786883 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" event={"ID":"35655276-9bb5-4f99-b6c9-c218c1e8b7d2","Type":"ContainerStarted","Data":"5e2f7b996340a84e9b4bfdbf04fe0ce72f9e97955b23897a7a6398cf5e4ed571"} Nov 03 01:08:10.787312 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:10.787093 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:10.911403 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:10.911344 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" podStartSLOduration=2.017521289 podStartE2EDuration="2.91133066s" totalImagesPullingTime="893.809371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:08:10.909241201 +0000 UTC m=+3737.860181360" watchObservedRunningTime="2026-11-03 01:08:10.91133066 +0000 UTC m=+3737.862270818" Nov 03 01:08:13.409164 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:13.409129 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:16.798352 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:16.798323 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:08:18.202514 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:18.202482 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:19.854426 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:19.854392 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:22.859266 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:22.859228 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:25.083256 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:25.083225 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:08:26.852629 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:26.852595 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb"] Nov 03 01:08:26.853123 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:26.852835 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" podUID="c3a73031-e0ee-45a7-ab4d-8a947f9812a2" containerName="maas-api" containerID="cri-o://2affa1d592124b24c69422e0ae8e21be514e9be1041fe33052a9a375c9635108" gracePeriod=30 Nov 03 01:08:26.991453 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:26.991419 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:27.204938 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:08:27.204902 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:08:27.222098 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.222070 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:08:27.385203 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.385170 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-kube-api-access-rwpsw\" (UniqueName: \"kubernetes.io/projected/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-kube-api-access-rwpsw\") pod \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " Nov 03 01:08:27.385376 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.385323 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-openshift-service-ca\") pod \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " Nov 03 01:08:27.385376 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.385348 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-metrics-certs\") pod \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " Nov 03 01:08:27.385487 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.385469 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-maas-api-tls\") pod \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\" (UID: \"c3a73031-e0ee-45a7-ab4d-8a947f9812a2\") " Nov 03 01:08:27.385744 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.385718 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-openshift-service-ca" pod "c3a73031-e0ee-45a7-ab4d-8a947f9812a2" (UID: "c3a73031-e0ee-45a7-ab4d-8a947f9812a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:08:27.386067 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.386042 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:27.387650 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.387627 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-metrics-certs" pod "c3a73031-e0ee-45a7-ab4d-8a947f9812a2" (UID: "c3a73031-e0ee-45a7-ab4d-8a947f9812a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:08:27.388134 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.388116 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-maas-api-tls" pod "c3a73031-e0ee-45a7-ab4d-8a947f9812a2" (UID: "c3a73031-e0ee-45a7-ab4d-8a947f9812a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:08:27.388249 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.388140 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-kube-api-access-rwpsw" pod "c3a73031-e0ee-45a7-ab4d-8a947f9812a2" (UID: "c3a73031-e0ee-45a7-ab4d-8a947f9812a2"). InnerVolumeSpecName "kube-api-access-rwpsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:08:27.487714 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.487635 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:27.487714 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.487673 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:27.487714 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.487688 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwpsw\" (UniqueName: \"kubernetes.io/projected/c3a73031-e0ee-45a7-ab4d-8a947f9812a2-kube-api-access-rwpsw\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:27.885034 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.885002 6400 generic.go:356] "Generic (PLEG): container finished" podID="c3a73031-e0ee-45a7-ab4d-8a947f9812a2" containerID="2affa1d592124b24c69422e0ae8e21be514e9be1041fe33052a9a375c9635108" exitCode=0 Nov 03 01:08:27.885519 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.885055 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" event={"ID":"c3a73031-e0ee-45a7-ab4d-8a947f9812a2","Type":"ContainerDied","Data":"2affa1d592124b24c69422e0ae8e21be514e9be1041fe33052a9a375c9635108"} Nov 03 01:08:27.885519 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.885097 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" event={"ID":"c3a73031-e0ee-45a7-ab4d-8a947f9812a2","Type":"ContainerDied","Data":"254410fa3f3424e51bfbbb915e2e2495aabea6ee4fc6d9df862df803f8a1677d"} Nov 03 01:08:27.885519 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.885109 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb" Nov 03 01:08:27.885519 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.885130 6400 scope.go:122] "RemoveContainer" containerID="2affa1d592124b24c69422e0ae8e21be514e9be1041fe33052a9a375c9635108" Nov 03 01:08:27.896320 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.896300 6400 scope.go:122] "RemoveContainer" containerID="2affa1d592124b24c69422e0ae8e21be514e9be1041fe33052a9a375c9635108" Nov 03 01:08:27.896605 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:08:27.896586 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2affa1d592124b24c69422e0ae8e21be514e9be1041fe33052a9a375c9635108\": container with ID starting with 2affa1d592124b24c69422e0ae8e21be514e9be1041fe33052a9a375c9635108 not found: ID does not exist" containerID="2affa1d592124b24c69422e0ae8e21be514e9be1041fe33052a9a375c9635108" Nov 03 01:08:27.896671 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:27.896618 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2affa1d592124b24c69422e0ae8e21be514e9be1041fe33052a9a375c9635108"} err="failed to get container status \"2affa1d592124b24c69422e0ae8e21be514e9be1041fe33052a9a375c9635108\": rpc error: code = NotFound desc = could not find container \"2affa1d592124b24c69422e0ae8e21be514e9be1041fe33052a9a375c9635108\": container with ID starting with 2affa1d592124b24c69422e0ae8e21be514e9be1041fe33052a9a375c9635108 not found: ID does not exist" Nov 03 01:08:28.047907 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:28.047876 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb"] Nov 03 01:08:28.104868 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:28.099263 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cac9f3-685df65f85-fhmjb"] Nov 03 01:08:28.714859 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:28.714783 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:29.179602 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:29.179572 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a73031-e0ee-45a7-ab4d-8a947f9812a2" path="/var/lib/kubelet/pods/c3a73031-e0ee-45a7-ab4d-8a947f9812a2/volumes" Nov 03 01:08:33.543344 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:33.543303 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:35.551860 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:35.551804 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:35.788023 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:35.787984 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:37.187562 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:37.187515 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:37.793977 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:37.793720 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:38.535588 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:38.535553 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:38.710569 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:38.710535 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:43.406936 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:43.406901 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:44.738486 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:44.738450 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:44.914359 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:44.914316 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:48.563897 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:48.563856 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:51.017578 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:51.017541 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:53.422928 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:53.422852 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:57.153756 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:08:57.153712 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:08:58.136474 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:08:58.136429 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:01.569926 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:01.569811 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:09:02.819165 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:02.819130 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:06.760685 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:06.760651 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl"] Nov 03 01:09:06.761206 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:06.760901 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" podUID="1c4a08b1-b2ad-4a97-8925-4e971934937a" containerName="maas-api" containerID="cri-o://a5823d73090977dc1119cf72f6d5dfc208c9949aecd5ea1a81852a22cb7f68c1" gracePeriod=30 Nov 03 01:09:06.954744 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:06.954707 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:07.079593 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.079285 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:09:07.162324 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.162287 6400 generic.go:356] "Generic (PLEG): container finished" podID="1c4a08b1-b2ad-4a97-8925-4e971934937a" containerID="a5823d73090977dc1119cf72f6d5dfc208c9949aecd5ea1a81852a22cb7f68c1" exitCode=0 Nov 03 01:09:07.162507 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.162360 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" Nov 03 01:09:07.162507 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.162360 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" event={"ID":"1c4a08b1-b2ad-4a97-8925-4e971934937a","Type":"ContainerDied","Data":"a5823d73090977dc1119cf72f6d5dfc208c9949aecd5ea1a81852a22cb7f68c1"} Nov 03 01:09:07.162507 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.162481 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" event={"ID":"1c4a08b1-b2ad-4a97-8925-4e971934937a","Type":"ContainerDied","Data":"a8e49b1e814f73e034c767cf3dc66f9b121bc1d00e7b59cd163df0cc0b134ca5"} Nov 03 01:09:07.162507 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.162502 6400 scope.go:122] "RemoveContainer" containerID="a5823d73090977dc1119cf72f6d5dfc208c9949aecd5ea1a81852a22cb7f68c1" Nov 03 01:09:07.173076 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.173055 6400 scope.go:122] "RemoveContainer" containerID="a5823d73090977dc1119cf72f6d5dfc208c9949aecd5ea1a81852a22cb7f68c1" Nov 03 01:09:07.173411 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:09:07.173384 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5823d73090977dc1119cf72f6d5dfc208c9949aecd5ea1a81852a22cb7f68c1\": container with ID starting with a5823d73090977dc1119cf72f6d5dfc208c9949aecd5ea1a81852a22cb7f68c1 not found: ID does not exist" containerID="a5823d73090977dc1119cf72f6d5dfc208c9949aecd5ea1a81852a22cb7f68c1" Nov 03 01:09:07.173503 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.173424 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5823d73090977dc1119cf72f6d5dfc208c9949aecd5ea1a81852a22cb7f68c1"} err="failed to get container status \"a5823d73090977dc1119cf72f6d5dfc208c9949aecd5ea1a81852a22cb7f68c1\": rpc error: code = NotFound desc = could not find container \"a5823d73090977dc1119cf72f6d5dfc208c9949aecd5ea1a81852a22cb7f68c1\": container with ID starting with a5823d73090977dc1119cf72f6d5dfc208c9949aecd5ea1a81852a22cb7f68c1 not found: ID does not exist" Nov 03 01:09:07.217889 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.217713 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-metrics-certs\") pod \"1c4a08b1-b2ad-4a97-8925-4e971934937a\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " Nov 03 01:09:07.218264 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.218226 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4a08b1-b2ad-4a97-8925-4e971934937a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4a08b1-b2ad-4a97-8925-4e971934937a-openshift-service-ca\") pod \"1c4a08b1-b2ad-4a97-8925-4e971934937a\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " Nov 03 01:09:07.218453 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.218433 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-maas-api-tls\") pod \"1c4a08b1-b2ad-4a97-8925-4e971934937a\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " Nov 03 01:09:07.218534 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.218487 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4a08b1-b2ad-4a97-8925-4e971934937a-kube-api-access-gmc2v\" (UniqueName: \"kubernetes.io/projected/1c4a08b1-b2ad-4a97-8925-4e971934937a-kube-api-access-gmc2v\") pod \"1c4a08b1-b2ad-4a97-8925-4e971934937a\" (UID: \"1c4a08b1-b2ad-4a97-8925-4e971934937a\") " Nov 03 01:09:07.220334 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.220301 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4a08b1-b2ad-4a97-8925-4e971934937a-openshift-service-ca" pod "1c4a08b1-b2ad-4a97-8925-4e971934937a" (UID: "1c4a08b1-b2ad-4a97-8925-4e971934937a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:07.258040 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.257997 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-metrics-certs" pod "1c4a08b1-b2ad-4a97-8925-4e971934937a" (UID: "1c4a08b1-b2ad-4a97-8925-4e971934937a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:09:07.258375 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.258338 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-maas-api-tls" pod "1c4a08b1-b2ad-4a97-8925-4e971934937a" (UID: "1c4a08b1-b2ad-4a97-8925-4e971934937a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:09:07.258766 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.258747 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4a08b1-b2ad-4a97-8925-4e971934937a-kube-api-access-gmc2v" pod "1c4a08b1-b2ad-4a97-8925-4e971934937a" (UID: "1c4a08b1-b2ad-4a97-8925-4e971934937a"). InnerVolumeSpecName "kube-api-access-gmc2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:07.319808 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.319772 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:07.319808 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.319803 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4a08b1-b2ad-4a97-8925-4e971934937a-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:07.319808 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.319813 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4a08b1-b2ad-4a97-8925-4e971934937a-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:07.320089 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.319858 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmc2v\" (UniqueName: \"kubernetes.io/projected/1c4a08b1-b2ad-4a97-8925-4e971934937a-kube-api-access-gmc2v\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:07.582435 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.582351 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl"] Nov 03 01:09:07.632019 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:07.631984 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl"] Nov 03 01:09:08.716595 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:08.716555 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:09.180250 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:09.180217 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4a08b1-b2ad-4a97-8925-4e971934937a" path="/var/lib/kubelet/pods/1c4a08b1-b2ad-4a97-8925-4e971934937a/volumes" Nov 03 01:09:09.976304 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:09.976258 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67b917-8994bc5b4-mlbdl" podUID="1c4a08b1-b2ad-4a97-8925-4e971934937a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.140:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:12.880156 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:12.880123 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:13.160799 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:13.160686 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:13.382445 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:13.382411 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:14.814051 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:14.814015 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:16.068482 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:16.068447 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:16.237356 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:16.237320 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:16.438421 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:16.438304 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:16.937845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:16.937304 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:17.587798 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:17.587762 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:17.756104 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:17.756070 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:19.321418 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:19.321378 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:19.546073 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:19.546033 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:20.034227 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:20.034192 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:20.195635 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:20.195583 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:20.809755 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:20.809721 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:23.002797 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:23.002760 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:24.898374 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:24.898336 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:27.153052 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:09:27.153016 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:09:29.610641 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:29.610601 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:33.710436 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:33.710385 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:33.870765 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:33.870708 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:34.429996 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:34.429950 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:35.822387 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:35.822353 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:36.290916 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:36.290880 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:36.463496 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:36.463460 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:37.666657 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:37.666620 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:39.297072 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:39.297028 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:42.124790 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:42.124738 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:42.419366 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:42.419292 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:42.606386 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:42.606349 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:42.774404 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:42.774327 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:44.053426 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:44.053376 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:44.256710 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:44.256670 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:44.726190 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:44.726157 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:48.776438 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:48.776392 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:48.985147 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:48.985110 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:49.692847 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:49.685415 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:50.335298 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:50.335248 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:52.649581 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:52.649538 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:54.350503 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:54.350467 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:55.514109 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:55.509431 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:55.680581 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:55.680533 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:56.797390 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:56.797352 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:56.971211 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:56.971177 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:09:57.152590 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:09:57.152502 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:09:57.443229 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:57.443109 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:57.616630 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:57.616595 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:58.774460 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:58.774425 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:58.950094 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:58.950054 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:59.312026 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:09:59.311987 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:03.326986 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.326952 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7"] Nov 03 01:10:03.327400 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.327181 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" podUID="98f1b4dc-d5fb-402b-a75d-37a558b05167" containerName="maas-api" containerID="cri-o://8f5af126f70e4302f861954e6cadc335734fdd454d8851c73de65f8e3fe09136" gracePeriod=30 Nov 03 01:10:03.467449 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.467414 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:03.609108 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.609017 6400 generic.go:356] "Generic (PLEG): container finished" podID="98f1b4dc-d5fb-402b-a75d-37a558b05167" containerID="8f5af126f70e4302f861954e6cadc335734fdd454d8851c73de65f8e3fe09136" exitCode=0 Nov 03 01:10:03.609267 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.609136 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" event={"ID":"98f1b4dc-d5fb-402b-a75d-37a558b05167","Type":"ContainerDied","Data":"8f5af126f70e4302f861954e6cadc335734fdd454d8851c73de65f8e3fe09136"} Nov 03 01:10:03.650008 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.649979 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:10:03.708657 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.708622 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f1b4dc-d5fb-402b-a75d-37a558b05167-kube-api-access-96dr8\" (UniqueName: \"kubernetes.io/projected/98f1b4dc-d5fb-402b-a75d-37a558b05167-kube-api-access-96dr8\") pod \"98f1b4dc-d5fb-402b-a75d-37a558b05167\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " Nov 03 01:10:03.708904 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.708688 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-maas-api-tls\") pod \"98f1b4dc-d5fb-402b-a75d-37a558b05167\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " Nov 03 01:10:03.708904 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.708806 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-metrics-certs\") pod \"98f1b4dc-d5fb-402b-a75d-37a558b05167\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " Nov 03 01:10:03.709042 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.708901 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98f1b4dc-d5fb-402b-a75d-37a558b05167-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98f1b4dc-d5fb-402b-a75d-37a558b05167-openshift-service-ca\") pod \"98f1b4dc-d5fb-402b-a75d-37a558b05167\" (UID: \"98f1b4dc-d5fb-402b-a75d-37a558b05167\") " Nov 03 01:10:03.709540 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.709510 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f1b4dc-d5fb-402b-a75d-37a558b05167-openshift-service-ca" pod "98f1b4dc-d5fb-402b-a75d-37a558b05167" (UID: "98f1b4dc-d5fb-402b-a75d-37a558b05167"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:03.711150 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.711124 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f1b4dc-d5fb-402b-a75d-37a558b05167-kube-api-access-96dr8" pod "98f1b4dc-d5fb-402b-a75d-37a558b05167" (UID: "98f1b4dc-d5fb-402b-a75d-37a558b05167"). InnerVolumeSpecName "kube-api-access-96dr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:03.711365 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.711347 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-maas-api-tls" pod "98f1b4dc-d5fb-402b-a75d-37a558b05167" (UID: "98f1b4dc-d5fb-402b-a75d-37a558b05167"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:10:03.711528 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.711511 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-metrics-certs" pod "98f1b4dc-d5fb-402b-a75d-37a558b05167" (UID: "98f1b4dc-d5fb-402b-a75d-37a558b05167"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:10:03.819841 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.819773 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96dr8\" (UniqueName: \"kubernetes.io/projected/98f1b4dc-d5fb-402b-a75d-37a558b05167-kube-api-access-96dr8\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:03.819841 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.819811 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:03.820124 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.819852 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98f1b4dc-d5fb-402b-a75d-37a558b05167-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:03.820124 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:03.819866 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98f1b4dc-d5fb-402b-a75d-37a558b05167-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:04.617792 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:04.617750 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" event={"ID":"98f1b4dc-d5fb-402b-a75d-37a558b05167","Type":"ContainerDied","Data":"ada5247e44ab91a8d95b30976801853667a961043ea44f83670662beb31ea3bc"} Nov 03 01:10:04.617792 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:04.617791 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7" Nov 03 01:10:04.618304 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:04.617834 6400 scope.go:122] "RemoveContainer" containerID="8f5af126f70e4302f861954e6cadc335734fdd454d8851c73de65f8e3fe09136" Nov 03 01:10:04.780176 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:04.780135 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7"] Nov 03 01:10:04.830021 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:04.829987 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4bfaef-7b8bd67b9c-pg2v7"] Nov 03 01:10:04.865486 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:04.865447 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:05.066302 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:05.066269 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:05.179062 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:05.179031 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98f1b4dc-d5fb-402b-a75d-37a558b05167" path="/var/lib/kubelet/pods/98f1b4dc-d5fb-402b-a75d-37a558b05167/volumes" Nov 03 01:10:05.312122 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:05.312089 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:08.696702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:08.696667 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:08.860466 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:08.860432 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:10.228367 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:10.228329 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:10.447973 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:10.447932 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:10.714014 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:10.711907 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd"] Nov 03 01:10:10.714014 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:10.712267 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerName="istio-proxy" containerID="cri-o://c58b30ea2585fc5bfec2e71b5b3c46fbcb2606a1d08eb8ced61cf8638037845d" gracePeriod=30 Nov 03 01:10:10.728288 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:10.728254 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:11.778312 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:11.778275 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:12.012443 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:12.012412 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:12.251610 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:12.251564 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:12.497792 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:12.497750 6400 patch_prober.go:29] interesting pod/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:10:12.498011 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:12.497850 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:10:15.359209 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:15.359172 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:16.395490 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.395459 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:10:16.482581 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.482538 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-kube-api-access-tzghh\" (UniqueName: \"kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-kube-api-access-tzghh\") pod \"098e3787-f743-4111-b1d0-9f8806ce3c48\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " Nov 03 01:10:16.482779 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.482625 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-certs\") pod \"098e3787-f743-4111-b1d0-9f8806ce3c48\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " Nov 03 01:10:16.482779 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.482659 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-credential-socket\") pod \"098e3787-f743-4111-b1d0-9f8806ce3c48\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " Nov 03 01:10:16.482779 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.482711 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-socket\") pod \"098e3787-f743-4111-b1d0-9f8806ce3c48\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " Nov 03 01:10:16.482974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.482790 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-token\" (UniqueName: \"kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-token\") pod \"098e3787-f743-4111-b1d0-9f8806ce3c48\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " Nov 03 01:10:16.482974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.482885 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-envoy\") pod \"098e3787-f743-4111-b1d0-9f8806ce3c48\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " Nov 03 01:10:16.482974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.482965 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-podinfo\") pod \"098e3787-f743-4111-b1d0-9f8806ce3c48\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " Nov 03 01:10:16.483136 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.483006 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098e3787-f743-4111-b1d0-9f8806ce3c48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098e3787-f743-4111-b1d0-9f8806ce3c48-istiod-ca-cert\") pod \"098e3787-f743-4111-b1d0-9f8806ce3c48\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " Nov 03 01:10:16.483136 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.483029 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-certs" pod "098e3787-f743-4111-b1d0-9f8806ce3c48" (UID: "098e3787-f743-4111-b1d0-9f8806ce3c48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:16.483136 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.483053 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-data\") pod \"098e3787-f743-4111-b1d0-9f8806ce3c48\" (UID: \"098e3787-f743-4111-b1d0-9f8806ce3c48\") " Nov 03 01:10:16.483136 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.483085 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-credential-socket" pod "098e3787-f743-4111-b1d0-9f8806ce3c48" (UID: "098e3787-f743-4111-b1d0-9f8806ce3c48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:16.483136 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.483093 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-socket" pod "098e3787-f743-4111-b1d0-9f8806ce3c48" (UID: "098e3787-f743-4111-b1d0-9f8806ce3c48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:16.483417 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.483395 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-data" pod "098e3787-f743-4111-b1d0-9f8806ce3c48" (UID: "098e3787-f743-4111-b1d0-9f8806ce3c48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:16.483553 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.483501 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098e3787-f743-4111-b1d0-9f8806ce3c48-istiod-ca-cert" pod "098e3787-f743-4111-b1d0-9f8806ce3c48" (UID: "098e3787-f743-4111-b1d0-9f8806ce3c48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:16.483553 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.483546 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:16.483744 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.483568 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:16.483744 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.483583 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:16.483744 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.483597 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:16.485352 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.485324 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-podinfo" pod "098e3787-f743-4111-b1d0-9f8806ce3c48" (UID: "098e3787-f743-4111-b1d0-9f8806ce3c48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:10:16.485448 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.485364 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-token" pod "098e3787-f743-4111-b1d0-9f8806ce3c48" (UID: "098e3787-f743-4111-b1d0-9f8806ce3c48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:16.485502 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.485448 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-kube-api-access-tzghh" pod "098e3787-f743-4111-b1d0-9f8806ce3c48" (UID: "098e3787-f743-4111-b1d0-9f8806ce3c48"). InnerVolumeSpecName "kube-api-access-tzghh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:16.485541 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.485500 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-envoy" pod "098e3787-f743-4111-b1d0-9f8806ce3c48" (UID: "098e3787-f743-4111-b1d0-9f8806ce3c48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:16.584559 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.584520 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:16.584559 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.584553 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:16.584559 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.584564 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098e3787-f743-4111-b1d0-9f8806ce3c48-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:16.584843 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.584575 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098e3787-f743-4111-b1d0-9f8806ce3c48-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:16.584843 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.584584 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzghh\" (UniqueName: \"kubernetes.io/projected/098e3787-f743-4111-b1d0-9f8806ce3c48-kube-api-access-tzghh\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:16.693814 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.693775 6400 generic.go:356] "Generic (PLEG): container finished" podID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerID="c58b30ea2585fc5bfec2e71b5b3c46fbcb2606a1d08eb8ced61cf8638037845d" exitCode=0 Nov 03 01:10:16.694002 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.693851 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" Nov 03 01:10:16.694002 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.693860 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" event={"ID":"098e3787-f743-4111-b1d0-9f8806ce3c48","Type":"ContainerDied","Data":"c58b30ea2585fc5bfec2e71b5b3c46fbcb2606a1d08eb8ced61cf8638037845d"} Nov 03 01:10:16.694002 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.693943 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd" event={"ID":"098e3787-f743-4111-b1d0-9f8806ce3c48","Type":"ContainerDied","Data":"bc0855468b31a86f1af46b9db5f12b10db71d9ef6f1a6f06cbd69ee7de3c3ef1"} Nov 03 01:10:16.694002 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.693966 6400 scope.go:122] "RemoveContainer" containerID="c58b30ea2585fc5bfec2e71b5b3c46fbcb2606a1d08eb8ced61cf8638037845d" Nov 03 01:10:16.704135 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.704116 6400 scope.go:122] "RemoveContainer" containerID="a6ce21300cea96924783beb60e826194a3644848b087c6c9acc3a32069cc09c1" Nov 03 01:10:16.713309 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.713289 6400 scope.go:122] "RemoveContainer" containerID="c58b30ea2585fc5bfec2e71b5b3c46fbcb2606a1d08eb8ced61cf8638037845d" Nov 03 01:10:16.713627 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:10:16.713605 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58b30ea2585fc5bfec2e71b5b3c46fbcb2606a1d08eb8ced61cf8638037845d\": container with ID starting with c58b30ea2585fc5bfec2e71b5b3c46fbcb2606a1d08eb8ced61cf8638037845d not found: ID does not exist" containerID="c58b30ea2585fc5bfec2e71b5b3c46fbcb2606a1d08eb8ced61cf8638037845d" Nov 03 01:10:16.713665 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.713638 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58b30ea2585fc5bfec2e71b5b3c46fbcb2606a1d08eb8ced61cf8638037845d"} err="failed to get container status \"c58b30ea2585fc5bfec2e71b5b3c46fbcb2606a1d08eb8ced61cf8638037845d\": rpc error: code = NotFound desc = could not find container \"c58b30ea2585fc5bfec2e71b5b3c46fbcb2606a1d08eb8ced61cf8638037845d\": container with ID starting with c58b30ea2585fc5bfec2e71b5b3c46fbcb2606a1d08eb8ced61cf8638037845d not found: ID does not exist" Nov 03 01:10:16.713665 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.713656 6400 scope.go:122] "RemoveContainer" containerID="a6ce21300cea96924783beb60e826194a3644848b087c6c9acc3a32069cc09c1" Nov 03 01:10:16.713942 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:10:16.713923 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ce21300cea96924783beb60e826194a3644848b087c6c9acc3a32069cc09c1\": container with ID starting with a6ce21300cea96924783beb60e826194a3644848b087c6c9acc3a32069cc09c1 not found: ID does not exist" containerID="a6ce21300cea96924783beb60e826194a3644848b087c6c9acc3a32069cc09c1" Nov 03 01:10:16.713987 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.713949 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ce21300cea96924783beb60e826194a3644848b087c6c9acc3a32069cc09c1"} err="failed to get container status \"a6ce21300cea96924783beb60e826194a3644848b087c6c9acc3a32069cc09c1\": rpc error: code = NotFound desc = could not find container \"a6ce21300cea96924783beb60e826194a3644848b087c6c9acc3a32069cc09c1\": container with ID starting with a6ce21300cea96924783beb60e826194a3644848b087c6c9acc3a32069cc09c1 not found: ID does not exist" Nov 03 01:10:16.858484 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.858446 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd"] Nov 03 01:10:16.864974 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:16.864943 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4bfaef-openshift-default-6977b685fc-9kzpd"] Nov 03 01:10:17.179492 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:17.179405 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" path="/var/lib/kubelet/pods/098e3787-f743-4111-b1d0-9f8806ce3c48/volumes" Nov 03 01:10:18.087872 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:18.087813 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:18.811257 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:18.811223 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:19.086880 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:19.086787 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:19.265185 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:19.265142 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:19.461897 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:19.461779 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:20.118954 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:20.118921 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:20.541169 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:20.541132 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:24.913458 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:24.913423 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:25.490851 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:25.490800 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:25.753945 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:25.753841 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:27.152967 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:10:27.152928 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:10:29.598312 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:29.598271 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:34.279850 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:34.279742 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:38.943024 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:38.942989 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:43.072502 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:43.072422 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:44.903842 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:44.903544 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:49.654046 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:49.654007 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:51.640493 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:51.640447 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:51.849410 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:51.849380 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:53.170550 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:53.170510 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:53.689061 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:53.689029 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:54.173711 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:54.173665 6400 scope.go:122] "RemoveContainer" containerID="2a770f3f669833d4146ae25433caaaa6303060db0a9d5a29643c3e5596a38926" Nov 03 01:10:54.521075 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:54.520951 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:54.756942 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:54.756904 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:57.153306 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:10:57.153268 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:10:59.423454 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:10:59.423414 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:00.811560 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:00.811522 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:01.329207 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:01.329171 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:01.480088 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:01.480054 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:04.615186 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:04.615136 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:05.975917 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:05.975876 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:06.190232 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:06.190193 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:06.714305 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:06.714254 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:09.326941 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:09.326905 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:14.013971 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:14.013924 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:16.892970 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:16.892935 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:18.770957 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:18.770918 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:20.877663 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:20.877628 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:23.122497 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:23.122457 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:11:23.512157 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:23.512052 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:24.791503 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:24.791470 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:27.152969 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:11:27.152930 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:11:28.256251 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:28.256204 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:33.024950 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:33.024911 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:37.833083 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:37.833041 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:42.616069 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:42.616033 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:47.343555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:47.343506 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:52.004490 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:52.004452 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:53.605641 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:53.605598 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:56.745343 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:56.745298 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:57.152971 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:11:57.152935 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:11:59.679681 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:11:59.679643 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:00.799529 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:00.799494 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh"] Nov 03 01:12:00.800118 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:00.799796 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" podUID="35655276-9bb5-4f99-b6c9-c218c1e8b7d2" containerName="maas-api" containerID="cri-o://5e2f7b996340a84e9b4bfdbf04fe0ce72f9e97955b23897a7a6398cf5e4ed571" gracePeriod=30 Nov 03 01:12:00.927976 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:00.927931 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:01.148366 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.148333 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:12:01.273815 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.273722 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-maas-api-tls\") pod \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " Nov 03 01:12:01.274028 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.273861 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-openshift-service-ca\") pod \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " Nov 03 01:12:01.274028 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.273922 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-metrics-certs\") pod \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " Nov 03 01:12:01.274175 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.274076 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-kube-api-access-td5kj\" (UniqueName: \"kubernetes.io/projected/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-kube-api-access-td5kj\") pod \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\" (UID: \"35655276-9bb5-4f99-b6c9-c218c1e8b7d2\") " Nov 03 01:12:01.274929 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.274898 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-openshift-service-ca" pod "35655276-9bb5-4f99-b6c9-c218c1e8b7d2" (UID: "35655276-9bb5-4f99-b6c9-c218c1e8b7d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:12:01.276388 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.276364 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-maas-api-tls" pod "35655276-9bb5-4f99-b6c9-c218c1e8b7d2" (UID: "35655276-9bb5-4f99-b6c9-c218c1e8b7d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:12:01.276964 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.276936 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-kube-api-access-td5kj" pod "35655276-9bb5-4f99-b6c9-c218c1e8b7d2" (UID: "35655276-9bb5-4f99-b6c9-c218c1e8b7d2"). InnerVolumeSpecName "kube-api-access-td5kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:12:01.279861 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.277950 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-metrics-certs" pod "35655276-9bb5-4f99-b6c9-c218c1e8b7d2" (UID: "35655276-9bb5-4f99-b6c9-c218c1e8b7d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:12:01.322252 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.322214 6400 generic.go:356] "Generic (PLEG): container finished" podID="35655276-9bb5-4f99-b6c9-c218c1e8b7d2" containerID="5e2f7b996340a84e9b4bfdbf04fe0ce72f9e97955b23897a7a6398cf5e4ed571" exitCode=0 Nov 03 01:12:01.322432 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.322268 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" Nov 03 01:12:01.322432 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.322305 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" event={"ID":"35655276-9bb5-4f99-b6c9-c218c1e8b7d2","Type":"ContainerDied","Data":"5e2f7b996340a84e9b4bfdbf04fe0ce72f9e97955b23897a7a6398cf5e4ed571"} Nov 03 01:12:01.322432 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.322362 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh" event={"ID":"35655276-9bb5-4f99-b6c9-c218c1e8b7d2","Type":"ContainerDied","Data":"e6fce000f2dd09ed4c9d6fa739c5efbf2af074c41951935be688069b0a48815d"} Nov 03 01:12:01.322432 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.322384 6400 scope.go:122] "RemoveContainer" containerID="5e2f7b996340a84e9b4bfdbf04fe0ce72f9e97955b23897a7a6398cf5e4ed571" Nov 03 01:12:01.333508 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.333490 6400 scope.go:122] "RemoveContainer" containerID="5e2f7b996340a84e9b4bfdbf04fe0ce72f9e97955b23897a7a6398cf5e4ed571" Nov 03 01:12:01.333852 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:12:01.333809 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2f7b996340a84e9b4bfdbf04fe0ce72f9e97955b23897a7a6398cf5e4ed571\": container with ID starting with 5e2f7b996340a84e9b4bfdbf04fe0ce72f9e97955b23897a7a6398cf5e4ed571 not found: ID does not exist" containerID="5e2f7b996340a84e9b4bfdbf04fe0ce72f9e97955b23897a7a6398cf5e4ed571" Nov 03 01:12:01.333956 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.333860 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2f7b996340a84e9b4bfdbf04fe0ce72f9e97955b23897a7a6398cf5e4ed571"} err="failed to get container status \"5e2f7b996340a84e9b4bfdbf04fe0ce72f9e97955b23897a7a6398cf5e4ed571\": rpc error: code = NotFound desc = could not find container \"5e2f7b996340a84e9b4bfdbf04fe0ce72f9e97955b23897a7a6398cf5e4ed571\": container with ID starting with 5e2f7b996340a84e9b4bfdbf04fe0ce72f9e97955b23897a7a6398cf5e4ed571 not found: ID does not exist" Nov 03 01:12:01.388263 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.375477 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:01.388263 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.375513 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td5kj\" (UniqueName: \"kubernetes.io/projected/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-kube-api-access-td5kj\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:01.388263 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.375529 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:01.388263 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.375544 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35655276-9bb5-4f99-b6c9-c218c1e8b7d2-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:01.485766 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.485731 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh"] Nov 03 01:12:01.489717 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:01.489689 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b47e01-6764978c96-hhjgh"] Nov 03 01:12:02.705978 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:02.705945 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:03.178816 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:03.178785 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35655276-9bb5-4f99-b6c9-c218c1e8b7d2" path="/var/lib/kubelet/pods/35655276-9bb5-4f99-b6c9-c218c1e8b7d2/volumes" Nov 03 01:12:04.996772 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:04.996730 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:05.161367 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:05.161332 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:06.592164 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:06.592126 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:07.119476 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:07.119444 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:07.676914 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:07.676878 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:11.156202 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.156161 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h"] Nov 03 01:12:11.156726 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.156702 6400 cpu_manager.go:414] "containerMap: removing container" podUID="35655276-9bb5-4f99-b6c9-c218c1e8b7d2" containerName="maas-api" Nov 03 01:12:11.156726 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.156722 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35655276-9bb5-4f99-b6c9-c218c1e8b7d2" containerName="maas-api" Nov 03 01:12:11.156949 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.156749 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1c4a08b1-b2ad-4a97-8925-4e971934937a" containerName="maas-api" Nov 03 01:12:11.156949 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.156780 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4a08b1-b2ad-4a97-8925-4e971934937a" containerName="maas-api" Nov 03 01:12:11.156949 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.156796 6400 cpu_manager.go:414] "containerMap: removing container" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerName="istio-proxy" Nov 03 01:12:11.156949 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.156804 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerName="istio-proxy" Nov 03 01:12:11.156949 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.156831 6400 cpu_manager.go:414] "containerMap: removing container" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerName="istio-proxy" Nov 03 01:12:11.156949 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.156839 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerName="istio-proxy" Nov 03 01:12:11.156949 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.156858 6400 cpu_manager.go:414] "containerMap: removing container" podUID="98f1b4dc-d5fb-402b-a75d-37a558b05167" containerName="maas-api" Nov 03 01:12:11.156949 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.156868 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f1b4dc-d5fb-402b-a75d-37a558b05167" containerName="maas-api" Nov 03 01:12:11.156949 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.156911 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c3a73031-e0ee-45a7-ab4d-8a947f9812a2" containerName="maas-api" Nov 03 01:12:11.156949 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.156919 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a73031-e0ee-45a7-ab4d-8a947f9812a2" containerName="maas-api" Nov 03 01:12:11.157344 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.157066 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a73031-e0ee-45a7-ab4d-8a947f9812a2" containerName="maas-api" Nov 03 01:12:11.157344 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.157086 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="35655276-9bb5-4f99-b6c9-c218c1e8b7d2" containerName="maas-api" Nov 03 01:12:11.157344 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.157099 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4a08b1-b2ad-4a97-8925-4e971934937a" containerName="maas-api" Nov 03 01:12:11.157344 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.157109 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerName="istio-proxy" Nov 03 01:12:11.157344 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.157121 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="098e3787-f743-4111-b1d0-9f8806ce3c48" containerName="istio-proxy" Nov 03 01:12:11.157344 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.157130 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f1b4dc-d5fb-402b-a75d-37a558b05167" containerName="maas-api" Nov 03 01:12:11.161446 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.161428 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.215698 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.215238 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbhsn\" (UniqueName: \"kubernetes.io/projected/03e891a7-9472-4300-b9f0-b75a745280b9-kube-api-access-hbhsn\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.215698 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.215306 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-dshm\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.215698 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.215357 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e891a7-9472-4300-b9f0-b75a745280b9-tls-certs\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.215698 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.215393 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-tmp-dir\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.215698 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.215539 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-home\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.215698 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.215640 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-model-cache\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.264850 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.264795 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b2e795\"/\"kube-root-ca.crt\"" Nov 03 01:12:11.265102 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.264909 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b2e795\"/\"openshift-service-ca.crt\"" Nov 03 01:12:11.265102 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.264929 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b2e795\"/\"sub-test-model-b2e795-kserve-self-signed-certs\"" Nov 03 01:12:11.265102 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.264947 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b2e795\"/\"default-dockercfg-vn8vg\"" Nov 03 01:12:11.267099 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.267070 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h"] Nov 03 01:12:11.278136 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.278100 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:11.337845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.332402 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbhsn\" (UniqueName: \"kubernetes.io/projected/03e891a7-9472-4300-b9f0-b75a745280b9-kube-api-access-hbhsn\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.337845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.332463 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-dshm\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.337845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.332520 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e891a7-9472-4300-b9f0-b75a745280b9-tls-certs\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.337845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.332557 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-tmp-dir\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.337845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.332618 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-home\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.337845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.332707 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-model-cache\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.337845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.333238 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-model-cache\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.341228 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.341163 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-dshm\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.341652 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.341628 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-tmp-dir\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.341930 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.341876 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-home\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.350047 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.350019 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e891a7-9472-4300-b9f0-b75a745280b9-tls-certs\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.391013 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.390975 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbhsn\" (UniqueName: \"kubernetes.io/projected/03e891a7-9472-4300-b9f0-b75a745280b9-kube-api-access-hbhsn\") pod \"sub-test-model-b2e795-kserve-6758474ccc-5kl2h\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.480120 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.480023 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:11.870631 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:11.870604 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h"] Nov 03 01:12:11.872879 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:12:11.872854 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03e891a7_9472_4300_b9f0_b75a745280b9.slice/crio-ffe5dfc1033c96f1884c5bc37d6b9647e4fe181bd5fefdaead85d39096b5ccf2 WatchSource:0}: Error finding container ffe5dfc1033c96f1884c5bc37d6b9647e4fe181bd5fefdaead85d39096b5ccf2: Status 404 returned error can't find the container with id ffe5dfc1033c96f1884c5bc37d6b9647e4fe181bd5fefdaead85d39096b5ccf2 Nov 03 01:12:12.385607 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:12.385570 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" event={"ID":"03e891a7-9472-4300-b9f0-b75a745280b9","Type":"ContainerStarted","Data":"cc0a340e21429cb293e76412e99300a393466d7d521eb0bf9d97e942eb2baf8d"} Nov 03 01:12:12.385607 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:12.385613 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" event={"ID":"03e891a7-9472-4300-b9f0-b75a745280b9","Type":"ContainerStarted","Data":"ffe5dfc1033c96f1884c5bc37d6b9647e4fe181bd5fefdaead85d39096b5ccf2"} Nov 03 01:12:12.480211 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:12.480173 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:12.527667 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:12.527634 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:12.544866 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:12.544776 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" podStartSLOduration=1.544754763 podStartE2EDuration="1.544754763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:12:12.54341825 +0000 UTC m=+3979.494358410" watchObservedRunningTime="2026-11-03 01:12:12.544754763 +0000 UTC m=+3979.495694921" Nov 03 01:12:12.746361 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:12.746275 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:12.899640 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:12.899604 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:17.496935 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:17.496894 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:21.235671 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:21.235630 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:21.480448 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:21.480414 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:21.493679 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:21.493599 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:22.203477 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:22.203434 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:22.451972 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:22.451943 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:12:25.397506 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:25.397469 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:26.901686 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:26.901652 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:27.153725 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:12:27.153630 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:12:31.712285 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:31.712247 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:33.715795 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:33.715750 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:33.942691 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:33.942656 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:12:36.742450 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:36.742411 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:37.041743 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:37.041708 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:39.724460 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:39.724427 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:41.506081 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:41.506046 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:46.236302 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:46.236258 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:50.892730 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:50.892693 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:55.640718 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:12:55.640679 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:57.154215 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:12:57.154176 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:13:00.340946 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:00.340914 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:05.006012 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:05.005976 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:09.799650 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:09.799606 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:14.507447 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:14.507412 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:19.207631 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:19.207595 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:21.571909 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:21.571873 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:23.992989 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:23.992949 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:24.230349 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:24.230314 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:24.823196 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:24.823159 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:24.909847 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:24.909365 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59"] Nov 03 01:13:24.920419 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:24.920389 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:24.943118 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:24.943083 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59"] Nov 03 01:13:24.989326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:24.989295 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-904a34\"/\"openshift-service-ca.crt\"" Nov 03 01:13:24.989326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:24.989296 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-904a34\"/\"kube-root-ca.crt\"" Nov 03 01:13:25.060846 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.056157 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-904a34\"/\"sub-test-model-904a34-kserve-self-signed-certs\"" Nov 03 01:13:25.060846 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.056294 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-904a34\"/\"default-dockercfg-txv2k\"" Nov 03 01:13:25.064665 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.064592 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:25.094277 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.094187 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-model-cache\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.094277 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.094243 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-dshm\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.094490 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.094361 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7683180-ca9e-4cd5-9956-60ea4baa8147-tls-certs\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.094490 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.094401 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-home\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.094490 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.094418 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-tmp-dir\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.094612 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.094574 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p67nw\" (UniqueName: \"kubernetes.io/projected/d7683180-ca9e-4cd5-9956-60ea4baa8147-kube-api-access-p67nw\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.195746 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.195697 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7683180-ca9e-4cd5-9956-60ea4baa8147-tls-certs\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.195991 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.195759 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-home\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.195991 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.195775 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-tmp-dir\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.195991 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.195844 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67nw\" (UniqueName: \"kubernetes.io/projected/d7683180-ca9e-4cd5-9956-60ea4baa8147-kube-api-access-p67nw\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.195991 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.195915 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-model-cache\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.195991 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.195960 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-dshm\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.196278 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.196223 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-home\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.196339 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.196311 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-model-cache\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.196420 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.196398 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-tmp-dir\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.198495 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.198460 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-dshm\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.198652 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.198532 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7683180-ca9e-4cd5-9956-60ea4baa8147-tls-certs\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.252924 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.252892 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p67nw\" (UniqueName: \"kubernetes.io/projected/d7683180-ca9e-4cd5-9956-60ea4baa8147-kube-api-access-p67nw\") pod \"sub-test-model-904a34-kserve-556796ccdc-ccm59\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.265780 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.265746 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:25.394329 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.394191 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:25.663412 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.663385 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59"] Nov 03 01:13:25.665523 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:13:25.665490 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7683180_ca9e_4cd5_9956_60ea4baa8147.slice/crio-000c90ff2a4f7a1d1d8e0558f4f01289a6298fe0dcf49f144c5052c45dd6c1d9 WatchSource:0}: Error finding container 000c90ff2a4f7a1d1d8e0558f4f01289a6298fe0dcf49f144c5052c45dd6c1d9: Status 404 returned error can't find the container with id 000c90ff2a4f7a1d1d8e0558f4f01289a6298fe0dcf49f144c5052c45dd6c1d9 Nov 03 01:13:25.879522 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.879481 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" event={"ID":"d7683180-ca9e-4cd5-9956-60ea4baa8147","Type":"ContainerStarted","Data":"77360d082d18eb855c41a87efee67a9611aa9be103e79fb1e9241a661b36cd18"} Nov 03 01:13:25.879731 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.879536 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" event={"ID":"d7683180-ca9e-4cd5-9956-60ea4baa8147","Type":"ContainerStarted","Data":"000c90ff2a4f7a1d1d8e0558f4f01289a6298fe0dcf49f144c5052c45dd6c1d9"} Nov 03 01:13:25.995449 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:25.995326 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" podStartSLOduration=1.995305847 podStartE2EDuration="1.995305847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:13:25.993614289 +0000 UTC m=+4052.944554448" watchObservedRunningTime="2026-11-03 01:13:25.995305847 +0000 UTC m=+4052.946246007" Nov 03 01:13:26.266207 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:26.266108 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:27.153444 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:13:27.153403 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:13:28.756355 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:28.756308 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:29.342268 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:29.342232 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:29.533116 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:29.533083 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:29.808043 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:29.808007 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:32.555404 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:32.555326 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:32.773942 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:32.773906 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:33.816905 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:33.816868 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:35.266182 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:35.266149 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:35.278999 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:35.278966 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:35.943591 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:35.943561 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:13:38.721176 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:38.721136 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:43.374845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:43.371601 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:48.112548 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:48.112495 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:52.804386 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:52.804345 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:57.152251 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:13:57.152213 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:13:57.523359 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:13:57.523280 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:01.888032 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:01.887995 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:14:02.252262 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:02.252175 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:04.289257 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:04.289222 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:04.455835 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:04.455799 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:05.237253 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:05.237216 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:06.925663 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:06.925624 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:11.733887 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:11.733845 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:12.377566 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:12.377526 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:16.415402 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:16.415369 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:17.947636 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:17.947592 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:18.199353 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:18.199255 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:21.105373 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:21.105337 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:21.967394 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:21.967350 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:23.996696 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:23.996652 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:25.769958 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:25.769576 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:27.152815 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:14:27.152770 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:14:29.109051 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:29.109012 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:30.521300 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:30.521262 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:31.465232 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:31.465193 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:34.949752 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:34.949712 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c"] Nov 03 01:14:34.953537 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:34.953519 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.057409 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.057375 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-aee0dd37-serving-cert\"" Nov 03 01:14:35.060885 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.060855 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c"] Nov 03 01:14:35.069866 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.069814 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-maas-api-tls\") pod \"maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.070038 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.069879 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2n4f\" (UniqueName: \"kubernetes.io/projected/48320114-ae7f-4b99-9aec-cc27e247fc4d-kube-api-access-l2n4f\") pod \"maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.070102 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.070035 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-metrics-certs\") pod \"maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.070222 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.070137 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48320114-ae7f-4b99-9aec-cc27e247fc4d-openshift-service-ca\") pod \"maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.166094 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.166058 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:35.171302 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.171280 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-maas-api-tls\") pod \"maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.171376 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.171313 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2n4f\" (UniqueName: \"kubernetes.io/projected/48320114-ae7f-4b99-9aec-cc27e247fc4d-kube-api-access-l2n4f\") pod \"maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.171376 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.171358 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-metrics-certs\") pod \"maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.171454 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.171399 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48320114-ae7f-4b99-9aec-cc27e247fc4d-openshift-service-ca\") pod \"maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.171454 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:14:35.171413 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-serving-cert: secret "maas-api-e2e-mt-aee0dd37-serving-cert" not found Nov 03 01:14:35.171533 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:14:35.171470 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-maas-api-tls podName:48320114-ae7f-4b99-9aec-cc27e247fc4d nodeName:}" failed. No retries permitted until 2026-11-03 01:14:35.671449742 +0000 UTC m=+4122.622389899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-maas-api-tls") pod "maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" (UID: "48320114-ae7f-4b99-9aec-cc27e247fc4d") : secret "maas-api-e2e-mt-aee0dd37-serving-cert" not found Nov 03 01:14:35.172172 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.172154 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48320114-ae7f-4b99-9aec-cc27e247fc4d-openshift-service-ca\") pod \"maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.174211 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.174190 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-metrics-certs\") pod \"maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.242422 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.242349 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2n4f\" (UniqueName: \"kubernetes.io/projected/48320114-ae7f-4b99-9aec-cc27e247fc4d-kube-api-access-l2n4f\") pod \"maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.375145 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.375110 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:35.546105 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.546064 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:35.678491 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.678452 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-maas-api-tls\") pod \"maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.681227 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.681201 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-maas-api-tls\") pod \"maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:35.867102 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:35.867014 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:36.243613 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:36.243587 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c"] Nov 03 01:14:36.246138 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:14:36.246093 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48320114_ae7f_4b99_9aec_cc27e247fc4d.slice/crio-34c1f7b9830fb3f0b23f04145f1a4c982274d0faf48ce2da10ba0c63b78f2995 WatchSource:0}: Error finding container 34c1f7b9830fb3f0b23f04145f1a4c982274d0faf48ce2da10ba0c63b78f2995: Status 404 returned error can't find the container with id 34c1f7b9830fb3f0b23f04145f1a4c982274d0faf48ce2da10ba0c63b78f2995 Nov 03 01:14:36.247659 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:36.247640 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:14:36.283588 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:36.283553 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" event={"ID":"48320114-ae7f-4b99-9aec-cc27e247fc4d","Type":"ContainerStarted","Data":"34c1f7b9830fb3f0b23f04145f1a4c982274d0faf48ce2da10ba0c63b78f2995"} Nov 03 01:14:37.291023 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:37.290983 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" event={"ID":"48320114-ae7f-4b99-9aec-cc27e247fc4d","Type":"ContainerStarted","Data":"1477b82b39e77bc22b5a003d2dc008a51f551f80ac91dedd212bcede241d87f4"} Nov 03 01:14:37.291416 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:37.291053 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:37.452301 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:37.452190 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" podStartSLOduration=2.535383195 podStartE2EDuration="3.452171112s" totalImagesPullingTime="916.787917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:37.45190097 +0000 UTC m=+4124.402841147" watchObservedRunningTime="2026-11-03 01:14:37.452171112 +0000 UTC m=+4124.403111272" Nov 03 01:14:37.668537 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:37.668502 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:40.114719 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:40.114685 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:43.302216 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:43.302178 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:14:44.913048 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:44.913009 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:48.768738 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:48.768697 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:14:49.821247 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:49.821218 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:52.437472 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:52.437432 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:54.522248 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:54.522217 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:57.152561 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:14:57.152523 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:14:59.307572 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:14:59.307538 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:00.223103 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.223067 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8"] Nov 03 01:15:00.226958 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.226941 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds"] Nov 03 01:15:00.227079 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.227059 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" Nov 03 01:15:00.230258 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.230242 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" Nov 03 01:15:00.243952 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.243932 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp"] Nov 03 01:15:00.248152 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.248136 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" Nov 03 01:15:00.254324 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.254299 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr"] Nov 03 01:15:00.258564 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.258544 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" Nov 03 01:15:00.288901 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.288877 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0dee74-1c25-495c-bf82-d6d155f42c2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp\" (UID: \"1b0dee74-1c25-495c-bf82-d6d155f42c2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" Nov 03 01:15:00.289055 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.288910 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r24l\" (UniqueName: \"kubernetes.io/projected/c5e04815-50f0-4f14-b1fd-73a120e09378-kube-api-access-5r24l\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds\" (UID: \"c5e04815-50f0-4f14-b1fd-73a120e09378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" Nov 03 01:15:00.289055 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.288932 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e04815-50f0-4f14-b1fd-73a120e09378-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds\" (UID: \"c5e04815-50f0-4f14-b1fd-73a120e09378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" Nov 03 01:15:00.289055 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.288954 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqqs5\" (UniqueName: \"kubernetes.io/projected/09687b53-7262-438c-a249-123aefe4ccf8-kube-api-access-tqqs5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8\" (UID: \"09687b53-7262-438c-a249-123aefe4ccf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" Nov 03 01:15:00.289187 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.289091 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847rn\" (UniqueName: \"kubernetes.io/projected/1b0dee74-1c25-495c-bf82-d6d155f42c2f-kube-api-access-847rn\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp\" (UID: \"1b0dee74-1c25-495c-bf82-d6d155f42c2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" Nov 03 01:15:00.289187 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.289145 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09687b53-7262-438c-a249-123aefe4ccf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8\" (UID: \"09687b53-7262-438c-a249-123aefe4ccf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" Nov 03 01:15:00.327408 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.327371 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8"] Nov 03 01:15:00.331117 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.331083 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds"] Nov 03 01:15:00.339192 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.339150 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:15:00.390225 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.390189 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847rn\" (UniqueName: \"kubernetes.io/projected/1b0dee74-1c25-495c-bf82-d6d155f42c2f-kube-api-access-847rn\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp\" (UID: \"1b0dee74-1c25-495c-bf82-d6d155f42c2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" Nov 03 01:15:00.390418 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.390240 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr\" (UID: \"4e42c2bc-a7d9-480b-b717-6411c0da5b8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" Nov 03 01:15:00.390418 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.390292 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09687b53-7262-438c-a249-123aefe4ccf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8\" (UID: \"09687b53-7262-438c-a249-123aefe4ccf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" Nov 03 01:15:00.390555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.390458 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qgh\" (UniqueName: \"kubernetes.io/projected/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-kube-api-access-49qgh\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr\" (UID: \"4e42c2bc-a7d9-480b-b717-6411c0da5b8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" Nov 03 01:15:00.390555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.390490 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0dee74-1c25-495c-bf82-d6d155f42c2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp\" (UID: \"1b0dee74-1c25-495c-bf82-d6d155f42c2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" Nov 03 01:15:00.390555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.390520 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r24l\" (UniqueName: \"kubernetes.io/projected/c5e04815-50f0-4f14-b1fd-73a120e09378-kube-api-access-5r24l\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds\" (UID: \"c5e04815-50f0-4f14-b1fd-73a120e09378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" Nov 03 01:15:00.390555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.390543 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e04815-50f0-4f14-b1fd-73a120e09378-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds\" (UID: \"c5e04815-50f0-4f14-b1fd-73a120e09378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" Nov 03 01:15:00.390779 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.390568 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqqs5\" (UniqueName: \"kubernetes.io/projected/09687b53-7262-438c-a249-123aefe4ccf8-kube-api-access-tqqs5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8\" (UID: \"09687b53-7262-438c-a249-123aefe4ccf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" Nov 03 01:15:00.391259 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.391233 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09687b53-7262-438c-a249-123aefe4ccf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8\" (UID: \"09687b53-7262-438c-a249-123aefe4ccf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" Nov 03 01:15:00.391448 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.391419 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0dee74-1c25-495c-bf82-d6d155f42c2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp\" (UID: \"1b0dee74-1c25-495c-bf82-d6d155f42c2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" Nov 03 01:15:00.391615 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.391596 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e04815-50f0-4f14-b1fd-73a120e09378-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds\" (UID: \"c5e04815-50f0-4f14-b1fd-73a120e09378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" Nov 03 01:15:00.400786 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.400724 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr"] Nov 03 01:15:00.404127 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.404099 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp"] Nov 03 01:15:00.446800 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.446770 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-847rn\" (UniqueName: \"kubernetes.io/projected/1b0dee74-1c25-495c-bf82-d6d155f42c2f-kube-api-access-847rn\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp\" (UID: \"1b0dee74-1c25-495c-bf82-d6d155f42c2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" Nov 03 01:15:00.447033 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.446975 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqqs5\" (UniqueName: \"kubernetes.io/projected/09687b53-7262-438c-a249-123aefe4ccf8-kube-api-access-tqqs5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8\" (UID: \"09687b53-7262-438c-a249-123aefe4ccf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" Nov 03 01:15:00.451248 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.451222 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r24l\" (UniqueName: \"kubernetes.io/projected/c5e04815-50f0-4f14-b1fd-73a120e09378-kube-api-access-5r24l\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds\" (UID: \"c5e04815-50f0-4f14-b1fd-73a120e09378\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" Nov 03 01:15:00.492104 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.492028 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qgh\" (UniqueName: \"kubernetes.io/projected/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-kube-api-access-49qgh\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr\" (UID: \"4e42c2bc-a7d9-480b-b717-6411c0da5b8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" Nov 03 01:15:00.492255 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.492123 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr\" (UID: \"4e42c2bc-a7d9-480b-b717-6411c0da5b8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" Nov 03 01:15:00.492738 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.492719 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr\" (UID: \"4e42c2bc-a7d9-480b-b717-6411c0da5b8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" Nov 03 01:15:00.541036 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.541004 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" Nov 03 01:15:00.548495 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.548479 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" Nov 03 01:15:00.562609 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.562582 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" Nov 03 01:15:00.582233 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.582194 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qgh\" (UniqueName: \"kubernetes.io/projected/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-kube-api-access-49qgh\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr\" (UID: \"4e42c2bc-a7d9-480b-b717-6411c0da5b8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" Nov 03 01:15:00.871308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.871280 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" Nov 03 01:15:00.929097 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.928762 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8"] Nov 03 01:15:00.943994 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:15:00.943957 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09687b53_7262_438c_a249_123aefe4ccf8.slice/crio-1c952c73facb782518a06f8a997c2cd92488fce8d2e56076b3f908bdeb2a678c WatchSource:0}: Error finding container 1c952c73facb782518a06f8a997c2cd92488fce8d2e56076b3f908bdeb2a678c: Status 404 returned error can't find the container with id 1c952c73facb782518a06f8a997c2cd92488fce8d2e56076b3f908bdeb2a678c Nov 03 01:15:00.977446 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.977420 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds"] Nov 03 01:15:00.980346 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:00.979972 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp"] Nov 03 01:15:00.982576 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:15:00.982544 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e04815_50f0_4f14_b1fd_73a120e09378.slice/crio-289e095186b59e594c770a00de7aacf46536cda3d2d87136fe376f4f3e408fa3 WatchSource:0}: Error finding container 289e095186b59e594c770a00de7aacf46536cda3d2d87136fe376f4f3e408fa3: Status 404 returned error can't find the container with id 289e095186b59e594c770a00de7aacf46536cda3d2d87136fe376f4f3e408fa3 Nov 03 01:15:01.246439 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:01.246411 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr"] Nov 03 01:15:01.249223 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:15:01.249189 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e42c2bc_a7d9_480b_b717_6411c0da5b8c.slice/crio-4f1190165ef3dc6765465fba6204e8698a38bc813c4c398d70a4de54e9a94f51 WatchSource:0}: Error finding container 4f1190165ef3dc6765465fba6204e8698a38bc813c4c398d70a4de54e9a94f51: Status 404 returned error can't find the container with id 4f1190165ef3dc6765465fba6204e8698a38bc813c4c398d70a4de54e9a94f51 Nov 03 01:15:01.469749 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:01.469658 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" event={"ID":"09687b53-7262-438c-a249-123aefe4ccf8","Type":"ContainerStarted","Data":"dbb74b4b511550a343720c75722da7797b73dff09c168d7679a119e76f49452d"} Nov 03 01:15:01.469749 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:01.469719 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" event={"ID":"09687b53-7262-438c-a249-123aefe4ccf8","Type":"ContainerStarted","Data":"1c952c73facb782518a06f8a997c2cd92488fce8d2e56076b3f908bdeb2a678c"} Nov 03 01:15:01.471218 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:01.471190 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" event={"ID":"4e42c2bc-a7d9-480b-b717-6411c0da5b8c","Type":"ContainerStarted","Data":"4f1190165ef3dc6765465fba6204e8698a38bc813c4c398d70a4de54e9a94f51"} Nov 03 01:15:01.473021 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:01.472993 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" event={"ID":"1b0dee74-1c25-495c-bf82-d6d155f42c2f","Type":"ContainerStarted","Data":"61fdf58cc6ed0e96cd44c22b3c634f26c07d7c49e8501b1b3c294d58c9ebd9bf"} Nov 03 01:15:01.473138 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:01.473042 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" event={"ID":"1b0dee74-1c25-495c-bf82-d6d155f42c2f","Type":"ContainerStarted","Data":"056b132635ed0837cac3495c4a7a840d2fde2a89e8d5698c6bda21644e33f264"} Nov 03 01:15:01.474367 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:01.474343 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" event={"ID":"c5e04815-50f0-4f14-b1fd-73a120e09378","Type":"ContainerStarted","Data":"289e095186b59e594c770a00de7aacf46536cda3d2d87136fe376f4f3e408fa3"} Nov 03 01:15:01.586276 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:01.586227 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" podStartSLOduration=1.586213715 podStartE2EDuration="1.586213715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:01.584255929 +0000 UTC m=+4148.535196086" watchObservedRunningTime="2026-11-03 01:15:01.586213715 +0000 UTC m=+4148.537153906" Nov 03 01:15:01.700013 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:01.699959 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" podStartSLOduration=1.699944922 podStartE2EDuration="1.699944922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:01.698100479 +0000 UTC m=+4148.649040637" watchObservedRunningTime="2026-11-03 01:15:01.699944922 +0000 UTC m=+4148.650885078" Nov 03 01:15:02.481994 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:02.481960 6400 generic.go:356] "Generic (PLEG): container finished" podID="09687b53-7262-438c-a249-123aefe4ccf8" containerID="dbb74b4b511550a343720c75722da7797b73dff09c168d7679a119e76f49452d" exitCode=60 Nov 03 01:15:02.482431 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:02.482046 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" event={"ID":"09687b53-7262-438c-a249-123aefe4ccf8","Type":"ContainerDied","Data":"dbb74b4b511550a343720c75722da7797b73dff09c168d7679a119e76f49452d"} Nov 03 01:15:02.482431 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:02.482308 6400 scope.go:122] "RemoveContainer" containerID="dbb74b4b511550a343720c75722da7797b73dff09c168d7679a119e76f49452d" Nov 03 01:15:02.483722 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:02.483701 6400 generic.go:356] "Generic (PLEG): container finished" podID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" containerID="47bb215e07ac7c52c914441df12477e0b3ff62e1177cfe94b967d61f9c01a815" exitCode=60 Nov 03 01:15:02.483900 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:02.483759 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" event={"ID":"4e42c2bc-a7d9-480b-b717-6411c0da5b8c","Type":"ContainerDied","Data":"47bb215e07ac7c52c914441df12477e0b3ff62e1177cfe94b967d61f9c01a815"} Nov 03 01:15:02.484063 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:02.484028 6400 scope.go:122] "RemoveContainer" containerID="47bb215e07ac7c52c914441df12477e0b3ff62e1177cfe94b967d61f9c01a815" Nov 03 01:15:02.485675 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:02.485501 6400 generic.go:356] "Generic (PLEG): container finished" podID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" containerID="61fdf58cc6ed0e96cd44c22b3c634f26c07d7c49e8501b1b3c294d58c9ebd9bf" exitCode=60 Nov 03 01:15:02.485675 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:02.485588 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" event={"ID":"1b0dee74-1c25-495c-bf82-d6d155f42c2f","Type":"ContainerDied","Data":"61fdf58cc6ed0e96cd44c22b3c634f26c07d7c49e8501b1b3c294d58c9ebd9bf"} Nov 03 01:15:02.485876 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:02.485751 6400 scope.go:122] "RemoveContainer" containerID="61fdf58cc6ed0e96cd44c22b3c634f26c07d7c49e8501b1b3c294d58c9ebd9bf" Nov 03 01:15:02.487273 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:02.487251 6400 generic.go:356] "Generic (PLEG): container finished" podID="c5e04815-50f0-4f14-b1fd-73a120e09378" containerID="e79f72815104dcaa55f293a8ca00a03d6eef334a2917b653722a9745261c1ae2" exitCode=60 Nov 03 01:15:02.487324 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:02.487272 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" event={"ID":"c5e04815-50f0-4f14-b1fd-73a120e09378","Type":"ContainerDied","Data":"e79f72815104dcaa55f293a8ca00a03d6eef334a2917b653722a9745261c1ae2"} Nov 03 01:15:02.487480 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:02.487467 6400 scope.go:122] "RemoveContainer" containerID="e79f72815104dcaa55f293a8ca00a03d6eef334a2917b653722a9745261c1ae2" Nov 03 01:15:03.498407 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:03.498378 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" event={"ID":"09687b53-7262-438c-a249-123aefe4ccf8","Type":"ContainerStarted","Data":"369b9a71a2d7f14430b6ade482c67c21975fef84a42cd4b1186f69809d2ba8c2"} Nov 03 01:15:03.501239 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:03.501214 6400 generic.go:356] "Generic (PLEG): container finished" podID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" containerID="0c27bfa200eeabf69b6dee2f6fd91dfc571b66c659ffbb7d1b6475e1d6528e22" exitCode=60 Nov 03 01:15:03.501373 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:03.501276 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" event={"ID":"4e42c2bc-a7d9-480b-b717-6411c0da5b8c","Type":"ContainerDied","Data":"0c27bfa200eeabf69b6dee2f6fd91dfc571b66c659ffbb7d1b6475e1d6528e22"} Nov 03 01:15:03.501373 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:03.501334 6400 scope.go:122] "RemoveContainer" containerID="47bb215e07ac7c52c914441df12477e0b3ff62e1177cfe94b967d61f9c01a815" Nov 03 01:15:03.501601 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:03.501582 6400 scope.go:122] "RemoveContainer" containerID="0c27bfa200eeabf69b6dee2f6fd91dfc571b66c659ffbb7d1b6475e1d6528e22" Nov 03 01:15:03.502063 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:03.501933 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr_odh-ai-gateway-infra(4e42c2bc-a7d9-480b-b717-6411c0da5b8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" podUID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" Nov 03 01:15:03.503748 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:03.503724 6400 generic.go:356] "Generic (PLEG): container finished" podID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" containerID="2407fe7b4ff020abf17b1d72aa3bb5b266cfc8420a901846147d7d6b60d12791" exitCode=60 Nov 03 01:15:03.504120 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:03.503810 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" event={"ID":"1b0dee74-1c25-495c-bf82-d6d155f42c2f","Type":"ContainerDied","Data":"2407fe7b4ff020abf17b1d72aa3bb5b266cfc8420a901846147d7d6b60d12791"} Nov 03 01:15:03.504199 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:03.504177 6400 scope.go:122] "RemoveContainer" containerID="2407fe7b4ff020abf17b1d72aa3bb5b266cfc8420a901846147d7d6b60d12791" Nov 03 01:15:03.504488 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:03.504468 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp_odh-ai-gateway-infra(1b0dee74-1c25-495c-bf82-d6d155f42c2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" podUID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" Nov 03 01:15:03.505976 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:03.505953 6400 generic.go:356] "Generic (PLEG): container finished" podID="c5e04815-50f0-4f14-b1fd-73a120e09378" containerID="1285fbf01e6e8664692a8a33c494f3fd2fbf8b405aa6cf7510ff08891da4f51e" exitCode=60 Nov 03 01:15:03.506078 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:03.506039 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" event={"ID":"c5e04815-50f0-4f14-b1fd-73a120e09378","Type":"ContainerDied","Data":"1285fbf01e6e8664692a8a33c494f3fd2fbf8b405aa6cf7510ff08891da4f51e"} Nov 03 01:15:03.506372 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:03.506354 6400 scope.go:122] "RemoveContainer" containerID="1285fbf01e6e8664692a8a33c494f3fd2fbf8b405aa6cf7510ff08891da4f51e" Nov 03 01:15:03.506675 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:03.506655 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds_odh-ai-gateway-infra(c5e04815-50f0-4f14-b1fd-73a120e09378)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" podUID="c5e04815-50f0-4f14-b1fd-73a120e09378" Nov 03 01:15:03.587341 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:03.587316 6400 scope.go:122] "RemoveContainer" containerID="61fdf58cc6ed0e96cd44c22b3c634f26c07d7c49e8501b1b3c294d58c9ebd9bf" Nov 03 01:15:03.600145 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:03.600125 6400 scope.go:122] "RemoveContainer" containerID="e79f72815104dcaa55f293a8ca00a03d6eef334a2917b653722a9745261c1ae2" Nov 03 01:15:04.148048 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:04.148006 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:04.514399 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:04.514252 6400 generic.go:356] "Generic (PLEG): container finished" podID="09687b53-7262-438c-a249-123aefe4ccf8" containerID="369b9a71a2d7f14430b6ade482c67c21975fef84a42cd4b1186f69809d2ba8c2" exitCode=60 Nov 03 01:15:04.514399 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:04.514324 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" event={"ID":"09687b53-7262-438c-a249-123aefe4ccf8","Type":"ContainerDied","Data":"369b9a71a2d7f14430b6ade482c67c21975fef84a42cd4b1186f69809d2ba8c2"} Nov 03 01:15:04.514399 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:04.514380 6400 scope.go:122] "RemoveContainer" containerID="dbb74b4b511550a343720c75722da7797b73dff09c168d7679a119e76f49452d" Nov 03 01:15:04.515017 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:04.514639 6400 scope.go:122] "RemoveContainer" containerID="369b9a71a2d7f14430b6ade482c67c21975fef84a42cd4b1186f69809d2ba8c2" Nov 03 01:15:04.515017 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:04.514966 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8_odh-ai-gateway-infra(09687b53-7262-438c-a249-123aefe4ccf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" podUID="09687b53-7262-438c-a249-123aefe4ccf8" Nov 03 01:15:04.516470 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:04.516455 6400 scope.go:122] "RemoveContainer" containerID="0c27bfa200eeabf69b6dee2f6fd91dfc571b66c659ffbb7d1b6475e1d6528e22" Nov 03 01:15:04.516748 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:04.516726 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr_odh-ai-gateway-infra(4e42c2bc-a7d9-480b-b717-6411c0da5b8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" podUID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" Nov 03 01:15:04.517939 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:04.517921 6400 scope.go:122] "RemoveContainer" containerID="2407fe7b4ff020abf17b1d72aa3bb5b266cfc8420a901846147d7d6b60d12791" Nov 03 01:15:04.518188 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:04.518167 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp_odh-ai-gateway-infra(1b0dee74-1c25-495c-bf82-d6d155f42c2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" podUID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" Nov 03 01:15:04.519410 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:04.519393 6400 scope.go:122] "RemoveContainer" containerID="1285fbf01e6e8664692a8a33c494f3fd2fbf8b405aa6cf7510ff08891da4f51e" Nov 03 01:15:04.519622 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:04.519604 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds_odh-ai-gateway-infra(c5e04815-50f0-4f14-b1fd-73a120e09378)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" podUID="c5e04815-50f0-4f14-b1fd-73a120e09378" Nov 03 01:15:05.530620 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:05.530593 6400 scope.go:122] "RemoveContainer" containerID="369b9a71a2d7f14430b6ade482c67c21975fef84a42cd4b1186f69809d2ba8c2" Nov 03 01:15:05.531042 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:05.530957 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8_odh-ai-gateway-infra(09687b53-7262-438c-a249-123aefe4ccf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" podUID="09687b53-7262-438c-a249-123aefe4ccf8" Nov 03 01:15:08.806947 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:08.806901 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:13.614650 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:13.614616 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:14.173070 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:14.173036 6400 scope.go:122] "RemoveContainer" containerID="1285fbf01e6e8664692a8a33c494f3fd2fbf8b405aa6cf7510ff08891da4f51e" Nov 03 01:15:14.173276 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:14.173160 6400 scope.go:122] "RemoveContainer" containerID="369b9a71a2d7f14430b6ade482c67c21975fef84a42cd4b1186f69809d2ba8c2" Nov 03 01:15:14.592628 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:14.592576 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" event={"ID":"09687b53-7262-438c-a249-123aefe4ccf8","Type":"ContainerStarted","Data":"190d01001b6f4917d2773620335f9de0e01b8f7f4c7b5bd41f2cfbe8933bcd36"} Nov 03 01:15:15.172810 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:15.172783 6400 scope.go:122] "RemoveContainer" containerID="2407fe7b4ff020abf17b1d72aa3bb5b266cfc8420a901846147d7d6b60d12791" Nov 03 01:15:15.600379 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:15.600346 6400 generic.go:356] "Generic (PLEG): container finished" podID="09687b53-7262-438c-a249-123aefe4ccf8" containerID="190d01001b6f4917d2773620335f9de0e01b8f7f4c7b5bd41f2cfbe8933bcd36" exitCode=60 Nov 03 01:15:15.600525 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:15.600421 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" event={"ID":"09687b53-7262-438c-a249-123aefe4ccf8","Type":"ContainerDied","Data":"190d01001b6f4917d2773620335f9de0e01b8f7f4c7b5bd41f2cfbe8933bcd36"} Nov 03 01:15:15.600525 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:15.600481 6400 scope.go:122] "RemoveContainer" containerID="369b9a71a2d7f14430b6ade482c67c21975fef84a42cd4b1186f69809d2ba8c2" Nov 03 01:15:15.600891 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:15.600873 6400 scope.go:122] "RemoveContainer" containerID="190d01001b6f4917d2773620335f9de0e01b8f7f4c7b5bd41f2cfbe8933bcd36" Nov 03 01:15:15.601186 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:15.601158 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8_odh-ai-gateway-infra(09687b53-7262-438c-a249-123aefe4ccf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" podUID="09687b53-7262-438c-a249-123aefe4ccf8" Nov 03 01:15:15.602384 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:15.602365 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" event={"ID":"1b0dee74-1c25-495c-bf82-d6d155f42c2f","Type":"ContainerStarted","Data":"9c586fd03b47921c050c21c8edb76b8d966bf4ffede0476b3dc0ff54bf9ae69f"} Nov 03 01:15:15.604047 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:15.604026 6400 generic.go:356] "Generic (PLEG): container finished" podID="c5e04815-50f0-4f14-b1fd-73a120e09378" containerID="a0f52d7dab0e726aaf5a0e4cdb3f3a0b7272487f74e77df86518090e5132ffb0" exitCode=60 Nov 03 01:15:15.604151 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:15.604093 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" event={"ID":"c5e04815-50f0-4f14-b1fd-73a120e09378","Type":"ContainerDied","Data":"a0f52d7dab0e726aaf5a0e4cdb3f3a0b7272487f74e77df86518090e5132ffb0"} Nov 03 01:15:15.604386 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:15.604373 6400 scope.go:122] "RemoveContainer" containerID="a0f52d7dab0e726aaf5a0e4cdb3f3a0b7272487f74e77df86518090e5132ffb0" Nov 03 01:15:15.604586 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:15.604570 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds_odh-ai-gateway-infra(c5e04815-50f0-4f14-b1fd-73a120e09378)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" podUID="c5e04815-50f0-4f14-b1fd-73a120e09378" Nov 03 01:15:15.612568 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:15.612549 6400 scope.go:122] "RemoveContainer" containerID="1285fbf01e6e8664692a8a33c494f3fd2fbf8b405aa6cf7510ff08891da4f51e" Nov 03 01:15:15.724625 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:15.724539 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8"] Nov 03 01:15:16.172248 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.172219 6400 scope.go:122] "RemoveContainer" containerID="0c27bfa200eeabf69b6dee2f6fd91dfc571b66c659ffbb7d1b6475e1d6528e22" Nov 03 01:15:16.613461 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.613431 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" event={"ID":"4e42c2bc-a7d9-480b-b717-6411c0da5b8c","Type":"ContainerStarted","Data":"83abfc6d554c9501914248f554db950efb298fade58a29d10e688df1a1fd751e"} Nov 03 01:15:16.615396 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.615373 6400 generic.go:356] "Generic (PLEG): container finished" podID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" containerID="9c586fd03b47921c050c21c8edb76b8d966bf4ffede0476b3dc0ff54bf9ae69f" exitCode=60 Nov 03 01:15:16.615511 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.615445 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" event={"ID":"1b0dee74-1c25-495c-bf82-d6d155f42c2f","Type":"ContainerDied","Data":"9c586fd03b47921c050c21c8edb76b8d966bf4ffede0476b3dc0ff54bf9ae69f"} Nov 03 01:15:16.615511 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.615490 6400 scope.go:122] "RemoveContainer" containerID="2407fe7b4ff020abf17b1d72aa3bb5b266cfc8420a901846147d7d6b60d12791" Nov 03 01:15:16.615813 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.615794 6400 scope.go:122] "RemoveContainer" containerID="9c586fd03b47921c050c21c8edb76b8d966bf4ffede0476b3dc0ff54bf9ae69f" Nov 03 01:15:16.616107 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:16.616083 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp_odh-ai-gateway-infra(1b0dee74-1c25-495c-bf82-d6d155f42c2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" podUID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" Nov 03 01:15:16.774403 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.774378 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" Nov 03 01:15:16.836651 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.836601 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" podStartSLOduration=16.836583284 podStartE2EDuration="16.836583284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:16.836042031 +0000 UTC m=+4163.786982215" watchObservedRunningTime="2026-11-03 01:15:16.836583284 +0000 UTC m=+4163.787523498" Nov 03 01:15:16.841600 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.841572 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds"] Nov 03 01:15:16.884889 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.884842 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09687b53-7262-438c-a249-123aefe4ccf8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09687b53-7262-438c-a249-123aefe4ccf8-maas-api-service-ca\") pod \"09687b53-7262-438c-a249-123aefe4ccf8\" (UID: \"09687b53-7262-438c-a249-123aefe4ccf8\") " Nov 03 01:15:16.885069 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.884929 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09687b53-7262-438c-a249-123aefe4ccf8-kube-api-access-tqqs5\" (UniqueName: \"kubernetes.io/projected/09687b53-7262-438c-a249-123aefe4ccf8-kube-api-access-tqqs5\") pod \"09687b53-7262-438c-a249-123aefe4ccf8\" (UID: \"09687b53-7262-438c-a249-123aefe4ccf8\") " Nov 03 01:15:16.885169 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.885147 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09687b53-7262-438c-a249-123aefe4ccf8-maas-api-service-ca" pod "09687b53-7262-438c-a249-123aefe4ccf8" (UID: "09687b53-7262-438c-a249-123aefe4ccf8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:16.885457 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.885438 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09687b53-7262-438c-a249-123aefe4ccf8-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:16.962018 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.961935 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp"] Nov 03 01:15:16.976511 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.976376 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09687b53-7262-438c-a249-123aefe4ccf8-kube-api-access-tqqs5" pod "09687b53-7262-438c-a249-123aefe4ccf8" (UID: "09687b53-7262-438c-a249-123aefe4ccf8"). InnerVolumeSpecName "kube-api-access-tqqs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:16.992852 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:16.992802 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqqs5\" (UniqueName: \"kubernetes.io/projected/09687b53-7262-438c-a249-123aefe4ccf8-kube-api-access-tqqs5\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:17.078037 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.078010 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" Nov 03 01:15:17.194849 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.194791 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e04815-50f0-4f14-b1fd-73a120e09378-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e04815-50f0-4f14-b1fd-73a120e09378-maas-api-service-ca\") pod \"c5e04815-50f0-4f14-b1fd-73a120e09378\" (UID: \"c5e04815-50f0-4f14-b1fd-73a120e09378\") " Nov 03 01:15:17.195009 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.194859 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e04815-50f0-4f14-b1fd-73a120e09378-kube-api-access-5r24l\" (UniqueName: \"kubernetes.io/projected/c5e04815-50f0-4f14-b1fd-73a120e09378-kube-api-access-5r24l\") pod \"c5e04815-50f0-4f14-b1fd-73a120e09378\" (UID: \"c5e04815-50f0-4f14-b1fd-73a120e09378\") " Nov 03 01:15:17.195141 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.195118 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e04815-50f0-4f14-b1fd-73a120e09378-maas-api-service-ca" pod "c5e04815-50f0-4f14-b1fd-73a120e09378" (UID: "c5e04815-50f0-4f14-b1fd-73a120e09378"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:17.195322 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.195298 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e04815-50f0-4f14-b1fd-73a120e09378-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:17.197059 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.197037 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e04815-50f0-4f14-b1fd-73a120e09378-kube-api-access-5r24l" pod "c5e04815-50f0-4f14-b1fd-73a120e09378" (UID: "c5e04815-50f0-4f14-b1fd-73a120e09378"). InnerVolumeSpecName "kube-api-access-5r24l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:17.297057 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.296786 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r24l\" (UniqueName: \"kubernetes.io/projected/c5e04815-50f0-4f14-b1fd-73a120e09378-kube-api-access-5r24l\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:17.625004 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.624921 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" Nov 03 01:15:17.625004 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.624934 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8" event={"ID":"09687b53-7262-438c-a249-123aefe4ccf8","Type":"ContainerDied","Data":"1c952c73facb782518a06f8a997c2cd92488fce8d2e56076b3f908bdeb2a678c"} Nov 03 01:15:17.625004 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.624996 6400 scope.go:122] "RemoveContainer" containerID="190d01001b6f4917d2773620335f9de0e01b8f7f4c7b5bd41f2cfbe8933bcd36" Nov 03 01:15:17.626909 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.626880 6400 generic.go:356] "Generic (PLEG): container finished" podID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" containerID="83abfc6d554c9501914248f554db950efb298fade58a29d10e688df1a1fd751e" exitCode=60 Nov 03 01:15:17.627042 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.626964 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" event={"ID":"4e42c2bc-a7d9-480b-b717-6411c0da5b8c","Type":"ContainerDied","Data":"83abfc6d554c9501914248f554db950efb298fade58a29d10e688df1a1fd751e"} Nov 03 01:15:17.627357 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.627322 6400 scope.go:122] "RemoveContainer" containerID="83abfc6d554c9501914248f554db950efb298fade58a29d10e688df1a1fd751e" Nov 03 01:15:17.627630 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:17.627606 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr_odh-ai-gateway-infra(4e42c2bc-a7d9-480b-b717-6411c0da5b8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" podUID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" Nov 03 01:15:17.630203 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.630185 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" Nov 03 01:15:17.630291 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.630231 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds" event={"ID":"c5e04815-50f0-4f14-b1fd-73a120e09378","Type":"ContainerDied","Data":"289e095186b59e594c770a00de7aacf46536cda3d2d87136fe376f4f3e408fa3"} Nov 03 01:15:17.640079 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.640008 6400 scope.go:122] "RemoveContainer" containerID="0c27bfa200eeabf69b6dee2f6fd91dfc571b66c659ffbb7d1b6475e1d6528e22" Nov 03 01:15:17.656967 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.656946 6400 scope.go:122] "RemoveContainer" containerID="a0f52d7dab0e726aaf5a0e4cdb3f3a0b7272487f74e77df86518090e5132ffb0" Nov 03 01:15:17.774564 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.774538 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" Nov 03 01:15:17.792831 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.792788 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds"] Nov 03 01:15:17.795333 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.795194 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894475-pqmds"] Nov 03 01:15:17.846329 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.846305 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr"] Nov 03 01:15:17.904170 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.904097 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0dee74-1c25-495c-bf82-d6d155f42c2f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0dee74-1c25-495c-bf82-d6d155f42c2f-maas-api-service-ca\") pod \"1b0dee74-1c25-495c-bf82-d6d155f42c2f\" (UID: \"1b0dee74-1c25-495c-bf82-d6d155f42c2f\") " Nov 03 01:15:17.904170 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.904136 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0dee74-1c25-495c-bf82-d6d155f42c2f-kube-api-access-847rn\" (UniqueName: \"kubernetes.io/projected/1b0dee74-1c25-495c-bf82-d6d155f42c2f-kube-api-access-847rn\") pod \"1b0dee74-1c25-495c-bf82-d6d155f42c2f\" (UID: \"1b0dee74-1c25-495c-bf82-d6d155f42c2f\") " Nov 03 01:15:17.904445 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.904421 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0dee74-1c25-495c-bf82-d6d155f42c2f-maas-api-service-ca" pod "1b0dee74-1c25-495c-bf82-d6d155f42c2f" (UID: "1b0dee74-1c25-495c-bf82-d6d155f42c2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:17.904531 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.904518 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0dee74-1c25-495c-bf82-d6d155f42c2f-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:17.906703 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:17.906679 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0dee74-1c25-495c-bf82-d6d155f42c2f-kube-api-access-847rn" pod "1b0dee74-1c25-495c-bf82-d6d155f42c2f" (UID: "1b0dee74-1c25-495c-bf82-d6d155f42c2f"). InnerVolumeSpecName "kube-api-access-847rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:18.005888 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.005810 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-847rn\" (UniqueName: \"kubernetes.io/projected/1b0dee74-1c25-495c-bf82-d6d155f42c2f-kube-api-access-847rn\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:18.130750 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.130717 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8"] Nov 03 01:15:18.134563 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.134529 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c8ce60-29894475-fg9r8"] Nov 03 01:15:18.327209 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.327172 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:18.639762 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.639737 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" Nov 03 01:15:18.640209 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.639763 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp" event={"ID":"1b0dee74-1c25-495c-bf82-d6d155f42c2f","Type":"ContainerDied","Data":"056b132635ed0837cac3495c4a7a840d2fde2a89e8d5698c6bda21644e33f264"} Nov 03 01:15:18.640209 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.639865 6400 scope.go:122] "RemoveContainer" containerID="9c586fd03b47921c050c21c8edb76b8d966bf4ffede0476b3dc0ff54bf9ae69f" Nov 03 01:15:18.780150 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.780128 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" Nov 03 01:15:18.861259 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.861212 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp"] Nov 03 01:15:18.868619 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.868593 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894475-j84rp"] Nov 03 01:15:18.916169 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.916088 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-kube-api-access-49qgh\" (UniqueName: \"kubernetes.io/projected/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-kube-api-access-49qgh\") pod \"4e42c2bc-a7d9-480b-b717-6411c0da5b8c\" (UID: \"4e42c2bc-a7d9-480b-b717-6411c0da5b8c\") " Nov 03 01:15:18.916169 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.916154 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-maas-api-service-ca\") pod \"4e42c2bc-a7d9-480b-b717-6411c0da5b8c\" (UID: \"4e42c2bc-a7d9-480b-b717-6411c0da5b8c\") " Nov 03 01:15:18.916504 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.916483 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-maas-api-service-ca" pod "4e42c2bc-a7d9-480b-b717-6411c0da5b8c" (UID: "4e42c2bc-a7d9-480b-b717-6411c0da5b8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:18.916630 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.916614 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-maas-api-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:18.918397 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:18.918371 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-kube-api-access-49qgh" pod "4e42c2bc-a7d9-480b-b717-6411c0da5b8c" (UID: "4e42c2bc-a7d9-480b-b717-6411c0da5b8c"). InnerVolumeSpecName "kube-api-access-49qgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:19.017474 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:19.017444 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49qgh\" (UniqueName: \"kubernetes.io/projected/4e42c2bc-a7d9-480b-b717-6411c0da5b8c-kube-api-access-49qgh\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.180082 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:19.180009 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09687b53-7262-438c-a249-123aefe4ccf8" path="/var/lib/kubelet/pods/09687b53-7262-438c-a249-123aefe4ccf8/volumes" Nov 03 01:15:19.180436 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:19.180422 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" path="/var/lib/kubelet/pods/1b0dee74-1c25-495c-bf82-d6d155f42c2f/volumes" Nov 03 01:15:19.180796 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:19.180782 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e04815-50f0-4f14-b1fd-73a120e09378" path="/var/lib/kubelet/pods/c5e04815-50f0-4f14-b1fd-73a120e09378/volumes" Nov 03 01:15:19.650426 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:19.650391 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" event={"ID":"4e42c2bc-a7d9-480b-b717-6411c0da5b8c","Type":"ContainerDied","Data":"4f1190165ef3dc6765465fba6204e8698a38bc813c4c398d70a4de54e9a94f51"} Nov 03 01:15:19.650971 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:19.650434 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr" Nov 03 01:15:19.650971 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:19.650457 6400 scope.go:122] "RemoveContainer" containerID="83abfc6d554c9501914248f554db950efb298fade58a29d10e688df1a1fd751e" Nov 03 01:15:19.813667 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:19.813635 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr"] Nov 03 01:15:19.863153 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:19.863122 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894475-fzhhr"] Nov 03 01:15:21.179078 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:21.179042 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" path="/var/lib/kubelet/pods/4e42c2bc-a7d9-480b-b717-6411c0da5b8c/volumes" Nov 03 01:15:22.296757 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:22.296717 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:23.175978 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:23.175945 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:27.152157 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:27.152122 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:15:27.793175 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:27.793134 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:28.252564 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:28.252474 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:28.452906 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:28.452865 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:32.474341 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:32.474246 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:37.135735 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:37.135705 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:38.265071 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:38.265008 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:38.435540 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:38.435508 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:41.831125 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:41.831090 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:44.293192 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:44.293157 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:15:46.627904 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:46.627864 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:48.557100 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:48.557059 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:51.425407 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:51.425372 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:52.461622 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:52.461590 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:53.356981 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:53.356945 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:53.494804 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:53.494765 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g"] Nov 03 01:15:53.578280 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:53.578240 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:53.761150 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:53.761057 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:54.063675 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:54.063637 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:56.500019 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:56.499984 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:57.152503 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:15:57.152464 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:15:59.718886 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:59.718849 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:59.919518 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:15:59.919473 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:01.236217 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:01.236186 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:02.245548 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:02.245508 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:06.016051 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:06.015930 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:06.182735 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:06.182697 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:07.153476 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:07.153437 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:07.345548 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:07.345515 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:08.509467 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.509397 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:08.521540 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.521496 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" podUID="89d21cbe-9619-4d67-bf52-105653fa5e73" containerName="main" containerID="cri-o://f57be52941981ae797a6406a134245e78fb20e061eb0609ce3d4505d030263eb" gracePeriod=45 Nov 03 01:16:08.799537 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.799513 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 01:16:08.923652 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.923618 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-tmp-dir\") pod \"89d21cbe-9619-4d67-bf52-105653fa5e73\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " Nov 03 01:16:08.923877 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.923693 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89d21cbe-9619-4d67-bf52-105653fa5e73-tls-certs\" (UniqueName: \"kubernetes.io/secret/89d21cbe-9619-4d67-bf52-105653fa5e73-tls-certs\") pod \"89d21cbe-9619-4d67-bf52-105653fa5e73\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " Nov 03 01:16:08.923877 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.923731 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-home\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-home\") pod \"89d21cbe-9619-4d67-bf52-105653fa5e73\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " Nov 03 01:16:08.923877 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.923758 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-dshm\") pod \"89d21cbe-9619-4d67-bf52-105653fa5e73\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " Nov 03 01:16:08.924720 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.923880 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d21cbe-9619-4d67-bf52-105653fa5e73-kube-api-access-7jgkc\" (UniqueName: \"kubernetes.io/projected/89d21cbe-9619-4d67-bf52-105653fa5e73-kube-api-access-7jgkc\") pod \"89d21cbe-9619-4d67-bf52-105653fa5e73\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " Nov 03 01:16:08.924720 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.923947 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-tmp-dir" pod "89d21cbe-9619-4d67-bf52-105653fa5e73" (UID: "89d21cbe-9619-4d67-bf52-105653fa5e73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.924720 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.924179 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-home" pod "89d21cbe-9619-4d67-bf52-105653fa5e73" (UID: "89d21cbe-9619-4d67-bf52-105653fa5e73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.924720 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.924191 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-model-cache" pod "89d21cbe-9619-4d67-bf52-105653fa5e73" (UID: "89d21cbe-9619-4d67-bf52-105653fa5e73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.924984 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.923955 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-model-cache\") pod \"89d21cbe-9619-4d67-bf52-105653fa5e73\" (UID: \"89d21cbe-9619-4d67-bf52-105653fa5e73\") " Nov 03 01:16:08.925405 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.925345 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.925405 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.925370 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.925405 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.925383 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.928454 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.928424 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d21cbe-9619-4d67-bf52-105653fa5e73-tls-certs" pod "89d21cbe-9619-4d67-bf52-105653fa5e73" (UID: "89d21cbe-9619-4d67-bf52-105653fa5e73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:16:08.928454 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.928436 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-dshm" pod "89d21cbe-9619-4d67-bf52-105653fa5e73" (UID: "89d21cbe-9619-4d67-bf52-105653fa5e73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.928454 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.928448 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d21cbe-9619-4d67-bf52-105653fa5e73-kube-api-access-7jgkc" pod "89d21cbe-9619-4d67-bf52-105653fa5e73" (UID: "89d21cbe-9619-4d67-bf52-105653fa5e73"). InnerVolumeSpecName "kube-api-access-7jgkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:08.929904 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.929874 6400 generic.go:356] "Generic (PLEG): container finished" podID="89d21cbe-9619-4d67-bf52-105653fa5e73" containerID="f57be52941981ae797a6406a134245e78fb20e061eb0609ce3d4505d030263eb" exitCode=0 Nov 03 01:16:08.930024 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.929922 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" Nov 03 01:16:08.930024 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.929943 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" event={"ID":"89d21cbe-9619-4d67-bf52-105653fa5e73","Type":"ContainerDied","Data":"f57be52941981ae797a6406a134245e78fb20e061eb0609ce3d4505d030263eb"} Nov 03 01:16:08.930024 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.929985 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g" event={"ID":"89d21cbe-9619-4d67-bf52-105653fa5e73","Type":"ContainerDied","Data":"d6256a4085b50b54ebeb1bafb1b6377c3d6625303a1a2d45c069334b603ea7b0"} Nov 03 01:16:08.930024 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.930017 6400 scope.go:122] "RemoveContainer" containerID="f57be52941981ae797a6406a134245e78fb20e061eb0609ce3d4505d030263eb" Nov 03 01:16:08.957235 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.957208 6400 scope.go:122] "RemoveContainer" containerID="f57be52941981ae797a6406a134245e78fb20e061eb0609ce3d4505d030263eb" Nov 03 01:16:08.957645 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:16:08.957626 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57be52941981ae797a6406a134245e78fb20e061eb0609ce3d4505d030263eb\": container with ID starting with f57be52941981ae797a6406a134245e78fb20e061eb0609ce3d4505d030263eb not found: ID does not exist" containerID="f57be52941981ae797a6406a134245e78fb20e061eb0609ce3d4505d030263eb" Nov 03 01:16:08.957711 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:08.957659 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57be52941981ae797a6406a134245e78fb20e061eb0609ce3d4505d030263eb"} err="failed to get container status \"f57be52941981ae797a6406a134245e78fb20e061eb0609ce3d4505d030263eb\": rpc error: code = NotFound desc = could not find container \"f57be52941981ae797a6406a134245e78fb20e061eb0609ce3d4505d030263eb\": container with ID starting with f57be52941981ae797a6406a134245e78fb20e061eb0609ce3d4505d030263eb not found: ID does not exist" Nov 03 01:16:09.026865 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:09.026763 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d21cbe-9619-4d67-bf52-105653fa5e73-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.026865 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:09.026803 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d21cbe-9619-4d67-bf52-105653fa5e73-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.026865 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:09.026840 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jgkc\" (UniqueName: \"kubernetes.io/projected/89d21cbe-9619-4d67-bf52-105653fa5e73-kube-api-access-7jgkc\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.096009 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:09.095976 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g"] Nov 03 01:16:09.097891 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:09.097865 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/premium-w0-ebeeb9-kserve-58d5bc6d66-t2r9g"] Nov 03 01:16:09.181551 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:09.181513 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d21cbe-9619-4d67-bf52-105653fa5e73" path="/var/lib/kubelet/pods/89d21cbe-9619-4d67-bf52-105653fa5e73/volumes" Nov 03 01:16:09.678129 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:09.677962 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:09.847175 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:09.847143 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:11.024674 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:11.024637 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:11.800652 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:11.800617 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:14.535718 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:14.535681 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:16.258665 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:16.258626 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:17.731149 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:17.731113 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:21.065416 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:21.060526 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:22.632760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:22.632719 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:25.667520 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:25.667482 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:27.151556 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:16:27.151521 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:16:30.368656 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:30.368614 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:30.886542 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:30.886506 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:31.072210 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:31.072168 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:31.916254 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:31.916212 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:35.081090 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:35.081055 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:37.800517 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:37.800479 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:38.021591 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:38.021558 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:39.828865 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:39.828808 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:42.999096 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:42.999061 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:16:44.574617 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:44.574578 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:49.520282 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:49.520246 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:54.219913 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:54.219877 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:57.152211 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:16:57.152158 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:16:58.954542 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:16:58.954508 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:03.591730 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:03.591693 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:08.259294 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:08.259258 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:12.285174 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.285136 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp"] Nov 03 01:17:12.285577 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.285343 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" podUID="5faaa2d3-41ef-42c6-a47c-8af1daeb5798" containerName="maas-api" containerID="cri-o://7fc22a35de1a3cb9af8ae0c018e233dcf8d550fb05392cc4e46fde76a993def3" gracePeriod=30 Nov 03 01:17:12.411291 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.411255 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:12.620628 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.620605 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 01:17:12.788435 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.788326 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-kube-api-access-sctnb\" (UniqueName: \"kubernetes.io/projected/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-kube-api-access-sctnb\") pod \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " Nov 03 01:17:12.788635 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.788502 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-maas-api-tls\") pod \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " Nov 03 01:17:12.788635 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.788571 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-metrics-certs\") pod \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " Nov 03 01:17:12.788754 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.788622 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-openshift-service-ca\") pod \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\" (UID: \"5faaa2d3-41ef-42c6-a47c-8af1daeb5798\") " Nov 03 01:17:12.789146 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.789100 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-openshift-service-ca" pod "5faaa2d3-41ef-42c6-a47c-8af1daeb5798" (UID: "5faaa2d3-41ef-42c6-a47c-8af1daeb5798"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:12.790947 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.790921 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-maas-api-tls" pod "5faaa2d3-41ef-42c6-a47c-8af1daeb5798" (UID: "5faaa2d3-41ef-42c6-a47c-8af1daeb5798"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:17:12.791074 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.790950 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-kube-api-access-sctnb" pod "5faaa2d3-41ef-42c6-a47c-8af1daeb5798" (UID: "5faaa2d3-41ef-42c6-a47c-8af1daeb5798"). InnerVolumeSpecName "kube-api-access-sctnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:12.791308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.791293 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-metrics-certs" pod "5faaa2d3-41ef-42c6-a47c-8af1daeb5798" (UID: "5faaa2d3-41ef-42c6-a47c-8af1daeb5798"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:17:12.890535 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.890457 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sctnb\" (UniqueName: \"kubernetes.io/projected/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-kube-api-access-sctnb\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:12.890535 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.890485 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:12.890535 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.890496 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:12.890535 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:12.890506 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5faaa2d3-41ef-42c6-a47c-8af1daeb5798-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:13.293270 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:13.293232 6400 generic.go:356] "Generic (PLEG): container finished" podID="5faaa2d3-41ef-42c6-a47c-8af1daeb5798" containerID="7fc22a35de1a3cb9af8ae0c018e233dcf8d550fb05392cc4e46fde76a993def3" exitCode=0 Nov 03 01:17:13.293742 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:13.293308 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" event={"ID":"5faaa2d3-41ef-42c6-a47c-8af1daeb5798","Type":"ContainerDied","Data":"7fc22a35de1a3cb9af8ae0c018e233dcf8d550fb05392cc4e46fde76a993def3"} Nov 03 01:17:13.293742 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:13.293365 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" event={"ID":"5faaa2d3-41ef-42c6-a47c-8af1daeb5798","Type":"ContainerDied","Data":"7ddb082774027cee0a3fe5192e646c2a873adb99472b95b90a4137604a058abe"} Nov 03 01:17:13.293742 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:13.293385 6400 scope.go:122] "RemoveContainer" containerID="7fc22a35de1a3cb9af8ae0c018e233dcf8d550fb05392cc4e46fde76a993def3" Nov 03 01:17:13.293742 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:13.293421 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp" Nov 03 01:17:13.310173 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:13.310151 6400 scope.go:122] "RemoveContainer" containerID="7fc22a35de1a3cb9af8ae0c018e233dcf8d550fb05392cc4e46fde76a993def3" Nov 03 01:17:13.310522 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:17:13.310498 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc22a35de1a3cb9af8ae0c018e233dcf8d550fb05392cc4e46fde76a993def3\": container with ID starting with 7fc22a35de1a3cb9af8ae0c018e233dcf8d550fb05392cc4e46fde76a993def3 not found: ID does not exist" containerID="7fc22a35de1a3cb9af8ae0c018e233dcf8d550fb05392cc4e46fde76a993def3" Nov 03 01:17:13.310596 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:13.310533 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc22a35de1a3cb9af8ae0c018e233dcf8d550fb05392cc4e46fde76a993def3"} err="failed to get container status \"7fc22a35de1a3cb9af8ae0c018e233dcf8d550fb05392cc4e46fde76a993def3\": rpc error: code = NotFound desc = could not find container \"7fc22a35de1a3cb9af8ae0c018e233dcf8d550fb05392cc4e46fde76a993def3\": container with ID starting with 7fc22a35de1a3cb9af8ae0c018e233dcf8d550fb05392cc4e46fde76a993def3 not found: ID does not exist" Nov 03 01:17:13.457884 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:13.457789 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp"] Nov 03 01:17:13.509675 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:13.509631 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4c8b1-b5758f94b-xqrrp"] Nov 03 01:17:13.561975 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:13.561941 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:15.031122 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:15.031084 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:15.179138 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:15.179107 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5faaa2d3-41ef-42c6-a47c-8af1daeb5798" path="/var/lib/kubelet/pods/5faaa2d3-41ef-42c6-a47c-8af1daeb5798/volumes" Nov 03 01:17:15.224365 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:15.224339 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:15.392723 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:15.392688 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:16.575768 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:16.575732 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:17.191224 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:17.191192 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:17.992045 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:17.992010 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:18.464539 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:18.464498 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:18.807120 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:18.807091 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:19.985929 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:19.985886 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:20.503731 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:20.503696 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:20.689618 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:20.689589 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:23.010272 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:23.010239 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:24.085229 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:24.085192 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:24.334831 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:24.334791 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:24.528067 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:24.527970 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:24.742749 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:24.742714 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:25.775216 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:25.775184 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:27.152245 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:17:27.152207 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:17:29.908450 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:29.908416 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:30.820553 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:30.820521 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:31.038060 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:31.038020 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:31.580991 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:31.580953 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:31.785978 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:31.785945 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:35.872428 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:35.872393 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:36.594000 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:36.593971 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:36.962052 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:36.961973 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:17:40.662281 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:40.662244 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:45.302101 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:45.302057 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:49.946772 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:49.946737 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:54.602134 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:54.602051 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:57.153108 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:17:57.153070 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:17:59.297324 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:17:59.297278 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:03.087508 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.087426 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:03.268572 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.268541 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c"] Nov 03 01:18:03.268767 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.268746 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" podUID="48320114-ae7f-4b99-9aec-cc27e247fc4d" containerName="maas-api" containerID="cri-o://1477b82b39e77bc22b5a003d2dc008a51f551f80ac91dedd212bcede241d87f4" gracePeriod=30 Nov 03 01:18:03.297307 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.297276 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" podUID="48320114-ae7f-4b99-9aec-cc27e247fc4d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.144:8443/health\": dial tcp 10.134.0.144:8443: connect: connection refused" Nov 03 01:18:03.420135 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.420098 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:03.595267 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.595244 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:18:03.611527 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.611491 6400 generic.go:356] "Generic (PLEG): container finished" podID="48320114-ae7f-4b99-9aec-cc27e247fc4d" containerID="1477b82b39e77bc22b5a003d2dc008a51f551f80ac91dedd212bcede241d87f4" exitCode=0 Nov 03 01:18:03.611688 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.611614 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" Nov 03 01:18:03.611688 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.611629 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" event={"ID":"48320114-ae7f-4b99-9aec-cc27e247fc4d","Type":"ContainerDied","Data":"1477b82b39e77bc22b5a003d2dc008a51f551f80ac91dedd212bcede241d87f4"} Nov 03 01:18:03.611768 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.611687 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c" event={"ID":"48320114-ae7f-4b99-9aec-cc27e247fc4d","Type":"ContainerDied","Data":"34c1f7b9830fb3f0b23f04145f1a4c982274d0faf48ce2da10ba0c63b78f2995"} Nov 03 01:18:03.611768 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.611712 6400 scope.go:122] "RemoveContainer" containerID="1477b82b39e77bc22b5a003d2dc008a51f551f80ac91dedd212bcede241d87f4" Nov 03 01:18:03.621361 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.621133 6400 scope.go:122] "RemoveContainer" containerID="1477b82b39e77bc22b5a003d2dc008a51f551f80ac91dedd212bcede241d87f4" Nov 03 01:18:03.621470 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:18:03.621428 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1477b82b39e77bc22b5a003d2dc008a51f551f80ac91dedd212bcede241d87f4\": container with ID starting with 1477b82b39e77bc22b5a003d2dc008a51f551f80ac91dedd212bcede241d87f4 not found: ID does not exist" containerID="1477b82b39e77bc22b5a003d2dc008a51f551f80ac91dedd212bcede241d87f4" Nov 03 01:18:03.621538 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.621477 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1477b82b39e77bc22b5a003d2dc008a51f551f80ac91dedd212bcede241d87f4"} err="failed to get container status \"1477b82b39e77bc22b5a003d2dc008a51f551f80ac91dedd212bcede241d87f4\": rpc error: code = NotFound desc = could not find container \"1477b82b39e77bc22b5a003d2dc008a51f551f80ac91dedd212bcede241d87f4\": container with ID starting with 1477b82b39e77bc22b5a003d2dc008a51f551f80ac91dedd212bcede241d87f4 not found: ID does not exist" Nov 03 01:18:03.711261 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.711173 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-maas-api-tls\") pod \"48320114-ae7f-4b99-9aec-cc27e247fc4d\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " Nov 03 01:18:03.711446 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.711279 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48320114-ae7f-4b99-9aec-cc27e247fc4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48320114-ae7f-4b99-9aec-cc27e247fc4d-openshift-service-ca\") pod \"48320114-ae7f-4b99-9aec-cc27e247fc4d\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " Nov 03 01:18:03.711446 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.711347 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-metrics-certs\") pod \"48320114-ae7f-4b99-9aec-cc27e247fc4d\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " Nov 03 01:18:03.711571 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.711521 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48320114-ae7f-4b99-9aec-cc27e247fc4d-kube-api-access-l2n4f\" (UniqueName: \"kubernetes.io/projected/48320114-ae7f-4b99-9aec-cc27e247fc4d-kube-api-access-l2n4f\") pod \"48320114-ae7f-4b99-9aec-cc27e247fc4d\" (UID: \"48320114-ae7f-4b99-9aec-cc27e247fc4d\") " Nov 03 01:18:03.711637 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.711612 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48320114-ae7f-4b99-9aec-cc27e247fc4d-openshift-service-ca" pod "48320114-ae7f-4b99-9aec-cc27e247fc4d" (UID: "48320114-ae7f-4b99-9aec-cc27e247fc4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:03.712076 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.712050 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48320114-ae7f-4b99-9aec-cc27e247fc4d-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:03.713735 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.713708 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-metrics-certs" pod "48320114-ae7f-4b99-9aec-cc27e247fc4d" (UID: "48320114-ae7f-4b99-9aec-cc27e247fc4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:03.714053 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.714023 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48320114-ae7f-4b99-9aec-cc27e247fc4d-kube-api-access-l2n4f" pod "48320114-ae7f-4b99-9aec-cc27e247fc4d" (UID: "48320114-ae7f-4b99-9aec-cc27e247fc4d"). InnerVolumeSpecName "kube-api-access-l2n4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:03.714053 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.714034 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-maas-api-tls" pod "48320114-ae7f-4b99-9aec-cc27e247fc4d" (UID: "48320114-ae7f-4b99-9aec-cc27e247fc4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:03.813424 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.813380 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:03.813424 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.813422 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2n4f\" (UniqueName: \"kubernetes.io/projected/48320114-ae7f-4b99-9aec-cc27e247fc4d-kube-api-access-l2n4f\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:03.813424 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:03.813435 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48320114-ae7f-4b99-9aec-cc27e247fc4d-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:04.032109 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:04.032021 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c"] Nov 03 01:18:04.035891 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:04.035865 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aee0dd37-f4d78dcb9-q548c"] Nov 03 01:18:05.103367 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:05.103330 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:05.180766 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:05.180706 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48320114-ae7f-4b99-9aec-cc27e247fc4d" path="/var/lib/kubelet/pods/48320114-ae7f-4b99-9aec-cc27e247fc4d/volumes" Nov 03 01:18:08.811845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:08.809360 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:09.066477 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:09.066390 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:09.936796 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:09.936762 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:10.333030 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:10.332988 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:10.806694 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:10.806663 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:11.569688 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:11.569651 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:15.029190 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:15.029154 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:17.372157 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:17.372124 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:18.520434 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:18.520393 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:18.675991 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:18.675929 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:19.120398 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.120360 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp"] Nov 03 01:18:19.120659 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.120632 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" podUID="a9fc68ce-c4aa-4e6f-b265-8802ee46de54" containerName="maas-api" containerID="cri-o://c124c7a27ebccdfcb1b71924de7026f48d5c79c3b8d3f8d97504b85d24472b6a" gracePeriod=30 Nov 03 01:18:19.247950 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.247918 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:19.453187 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.453157 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 01:18:19.517130 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.517066 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-maas-api-tls\") pod \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " Nov 03 01:18:19.517334 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.517188 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-kube-api-access-nf45h\" (UniqueName: \"kubernetes.io/projected/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-kube-api-access-nf45h\") pod \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " Nov 03 01:18:19.517334 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.517224 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-metrics-certs\") pod \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " Nov 03 01:18:19.517458 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.517342 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-openshift-service-ca\") pod \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\" (UID: \"a9fc68ce-c4aa-4e6f-b265-8802ee46de54\") " Nov 03 01:18:19.518196 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.518166 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-openshift-service-ca" pod "a9fc68ce-c4aa-4e6f-b265-8802ee46de54" (UID: "a9fc68ce-c4aa-4e6f-b265-8802ee46de54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:19.533665 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.533606 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-maas-api-tls" pod "a9fc68ce-c4aa-4e6f-b265-8802ee46de54" (UID: "a9fc68ce-c4aa-4e6f-b265-8802ee46de54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:19.542196 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.542095 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-kube-api-access-nf45h" pod "a9fc68ce-c4aa-4e6f-b265-8802ee46de54" (UID: "a9fc68ce-c4aa-4e6f-b265-8802ee46de54"). InnerVolumeSpecName "kube-api-access-nf45h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:19.563051 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.562991 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-metrics-certs" pod "a9fc68ce-c4aa-4e6f-b265-8802ee46de54" (UID: "a9fc68ce-c4aa-4e6f-b265-8802ee46de54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:19.618674 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.618633 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:19.618674 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.618674 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:19.618903 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.618689 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf45h\" (UniqueName: \"kubernetes.io/projected/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-kube-api-access-nf45h\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:19.618903 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.618703 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fc68ce-c4aa-4e6f-b265-8802ee46de54-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:19.701378 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.701288 6400 generic.go:356] "Generic (PLEG): container finished" podID="a9fc68ce-c4aa-4e6f-b265-8802ee46de54" containerID="c124c7a27ebccdfcb1b71924de7026f48d5c79c3b8d3f8d97504b85d24472b6a" exitCode=0 Nov 03 01:18:19.701378 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.701340 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" Nov 03 01:18:19.701576 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.701380 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" event={"ID":"a9fc68ce-c4aa-4e6f-b265-8802ee46de54","Type":"ContainerDied","Data":"c124c7a27ebccdfcb1b71924de7026f48d5c79c3b8d3f8d97504b85d24472b6a"} Nov 03 01:18:19.701576 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.701442 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp" event={"ID":"a9fc68ce-c4aa-4e6f-b265-8802ee46de54","Type":"ContainerDied","Data":"4ff71b97f05c10e96878ad2eedd5a3aa083a0b1dc5ac90c402b2f7235fdc2e8d"} Nov 03 01:18:19.701576 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.701471 6400 scope.go:122] "RemoveContainer" containerID="c124c7a27ebccdfcb1b71924de7026f48d5c79c3b8d3f8d97504b85d24472b6a" Nov 03 01:18:19.711672 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.711649 6400 scope.go:122] "RemoveContainer" containerID="c124c7a27ebccdfcb1b71924de7026f48d5c79c3b8d3f8d97504b85d24472b6a" Nov 03 01:18:19.711979 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:18:19.711959 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c124c7a27ebccdfcb1b71924de7026f48d5c79c3b8d3f8d97504b85d24472b6a\": container with ID starting with c124c7a27ebccdfcb1b71924de7026f48d5c79c3b8d3f8d97504b85d24472b6a not found: ID does not exist" containerID="c124c7a27ebccdfcb1b71924de7026f48d5c79c3b8d3f8d97504b85d24472b6a" Nov 03 01:18:19.712069 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.711988 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c124c7a27ebccdfcb1b71924de7026f48d5c79c3b8d3f8d97504b85d24472b6a"} err="failed to get container status \"c124c7a27ebccdfcb1b71924de7026f48d5c79c3b8d3f8d97504b85d24472b6a\": rpc error: code = NotFound desc = could not find container \"c124c7a27ebccdfcb1b71924de7026f48d5c79c3b8d3f8d97504b85d24472b6a\": container with ID starting with c124c7a27ebccdfcb1b71924de7026f48d5c79c3b8d3f8d97504b85d24472b6a not found: ID does not exist" Nov 03 01:18:19.864831 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.864788 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp"] Nov 03 01:18:19.869644 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:19.869612 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c87e2f-578594d48c-wl8lp"] Nov 03 01:18:21.152355 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:21.152324 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:21.196469 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:21.196430 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9fc68ce-c4aa-4e6f-b265-8802ee46de54" path="/var/lib/kubelet/pods/a9fc68ce-c4aa-4e6f-b265-8802ee46de54/volumes" Nov 03 01:18:21.519929 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:21.519842 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:18:22.906380 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:22.906348 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:23.316718 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:23.316683 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:25.795283 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:25.795246 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:27.151657 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:18:27.151623 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:18:29.604340 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:29.604306 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:29.767703 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:29.767673 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:30.574393 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:30.574357 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:31.142738 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:31.142708 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:31.577725 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:31.577694 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz"] Nov 03 01:18:31.577961 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:31.577906 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" containerID="cri-o://f0df182a0d526f7f0387a24f941c17ea08b91b8f667572fe81253a29dc5087b3" gracePeriod=30 Nov 03 01:18:31.617176 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:31.617137 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:32.054904 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:32.054868 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:35.809785 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:35.809748 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:35.882412 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:35.882375 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:18:35.882578 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:35.882453 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:18:36.800284 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.800223 6400 generic.go:356] "Generic (PLEG): container finished" podID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerID="f0df182a0d526f7f0387a24f941c17ea08b91b8f667572fe81253a29dc5087b3" exitCode=0 Nov 03 01:18:36.800284 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.800267 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" event={"ID":"4df412b7-1ae9-4f96-a92a-85e29370dabb","Type":"ContainerDied","Data":"f0df182a0d526f7f0387a24f941c17ea08b91b8f667572fe81253a29dc5087b3"} Nov 03 01:18:36.800532 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.800388 6400 scope.go:122] "RemoveContainer" containerID="b8bbf078a209709ab13ec512e64d21d1263f44d8a9bcad7b8bbae0e5b026411a" Nov 03 01:18:36.847694 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.847634 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 01:18:36.934138 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.934105 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-envoy\") pod \"4df412b7-1ae9-4f96-a92a-85e29370dabb\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " Nov 03 01:18:36.934319 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.934163 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-certs\") pod \"4df412b7-1ae9-4f96-a92a-85e29370dabb\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " Nov 03 01:18:36.934319 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.934191 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-token\" (UniqueName: \"kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-token\") pod \"4df412b7-1ae9-4f96-a92a-85e29370dabb\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " Nov 03 01:18:36.934319 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.934279 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-credential-socket\") pod \"4df412b7-1ae9-4f96-a92a-85e29370dabb\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " Nov 03 01:18:36.934502 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.934323 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df412b7-1ae9-4f96-a92a-85e29370dabb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df412b7-1ae9-4f96-a92a-85e29370dabb-istiod-ca-cert\") pod \"4df412b7-1ae9-4f96-a92a-85e29370dabb\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " Nov 03 01:18:36.934502 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.934355 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-podinfo\") pod \"4df412b7-1ae9-4f96-a92a-85e29370dabb\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " Nov 03 01:18:36.934502 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.934424 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-kube-api-access-w588v\" (UniqueName: \"kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-kube-api-access-w588v\") pod \"4df412b7-1ae9-4f96-a92a-85e29370dabb\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " Nov 03 01:18:36.934502 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.934492 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-certs" pod "4df412b7-1ae9-4f96-a92a-85e29370dabb" (UID: "4df412b7-1ae9-4f96-a92a-85e29370dabb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:36.934730 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.934505 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-data\") pod \"4df412b7-1ae9-4f96-a92a-85e29370dabb\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " Nov 03 01:18:36.934730 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.934549 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-socket\") pod \"4df412b7-1ae9-4f96-a92a-85e29370dabb\" (UID: \"4df412b7-1ae9-4f96-a92a-85e29370dabb\") " Nov 03 01:18:36.935226 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.934886 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-credential-socket" pod "4df412b7-1ae9-4f96-a92a-85e29370dabb" (UID: "4df412b7-1ae9-4f96-a92a-85e29370dabb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:36.935226 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.934921 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df412b7-1ae9-4f96-a92a-85e29370dabb-istiod-ca-cert" pod "4df412b7-1ae9-4f96-a92a-85e29370dabb" (UID: "4df412b7-1ae9-4f96-a92a-85e29370dabb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:36.935226 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.935089 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:36.935226 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.935185 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-data" pod "4df412b7-1ae9-4f96-a92a-85e29370dabb" (UID: "4df412b7-1ae9-4f96-a92a-85e29370dabb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:36.935226 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.935239 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df412b7-1ae9-4f96-a92a-85e29370dabb-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:36.935780 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.935256 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:36.935780 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.935272 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-socket" pod "4df412b7-1ae9-4f96-a92a-85e29370dabb" (UID: "4df412b7-1ae9-4f96-a92a-85e29370dabb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:36.936786 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.936763 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-token" pod "4df412b7-1ae9-4f96-a92a-85e29370dabb" (UID: "4df412b7-1ae9-4f96-a92a-85e29370dabb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:36.937021 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.936995 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-kube-api-access-w588v" pod "4df412b7-1ae9-4f96-a92a-85e29370dabb" (UID: "4df412b7-1ae9-4f96-a92a-85e29370dabb"). InnerVolumeSpecName "kube-api-access-w588v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:36.937380 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.937362 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-podinfo" pod "4df412b7-1ae9-4f96-a92a-85e29370dabb" (UID: "4df412b7-1ae9-4f96-a92a-85e29370dabb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:18:36.937380 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.937366 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-envoy" pod "4df412b7-1ae9-4f96-a92a-85e29370dabb" (UID: "4df412b7-1ae9-4f96-a92a-85e29370dabb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:36.990570 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.990528 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5"] Nov 03 01:18:36.996735 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:36.996702 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr"] Nov 03 01:18:37.036043 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:37.036012 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:37.036043 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:37.036038 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:37.036043 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:37.036048 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w588v\" (UniqueName: \"kubernetes.io/projected/4df412b7-1ae9-4f96-a92a-85e29370dabb-kube-api-access-w588v\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:37.036275 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:37.036058 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:37.036275 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:37.036067 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:37.036275 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:37.036074 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df412b7-1ae9-4f96-a92a-85e29370dabb-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:37.121946 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:37.121853 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:37.808114 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:37.808087 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" Nov 03 01:18:37.808304 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:37.808116 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz" event={"ID":"4df412b7-1ae9-4f96-a92a-85e29370dabb","Type":"ContainerDied","Data":"05ee826c84d9e2b546d38267ba86a0a70b47e101b72e32da30db223251ab7627"} Nov 03 01:18:37.808304 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:37.808180 6400 scope.go:122] "RemoveContainer" containerID="f0df182a0d526f7f0387a24f941c17ea08b91b8f667572fe81253a29dc5087b3" Nov 03 01:18:38.014446 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:38.014411 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz"] Nov 03 01:18:38.063807 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:38.063739 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c87e2f-openshift-default-8547849fb9-h26vz"] Nov 03 01:18:39.030268 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:39.030236 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:39.178606 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:39.178571 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" path="/var/lib/kubelet/pods/4df412b7-1ae9-4f96-a92a-85e29370dabb/volumes" Nov 03 01:18:39.867665 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:39.867621 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:41.608265 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:41.608230 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:45.283197 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:45.283157 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:45.993072 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:45.993039 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:46.150422 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:46.150365 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:46.361092 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:46.361050 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:46.605535 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:46.605502 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:46.805929 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:46.805896 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:47.891009 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:47.890973 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:48.480290 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:48.480246 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:49.233979 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:49.233950 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:50.911279 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:50.911243 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:51.980606 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:51.980557 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:52.011081 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.011037 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" podUID="eaf4988a-ac9f-446e-91ec-28ee63f944a7" containerName="main" containerID="cri-o://03e0dd81c9b62cca3890c26b7f8cf03aa607b5d7b936c6aee7aa6b245fb43050" gracePeriod=45 Nov 03 01:18:52.017163 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.016805 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" podUID="f60672ea-e415-4096-bf4f-4689d4194dbf" containerName="main" containerID="cri-o://4bb68f545c59da36c217afa86f128b14d7fa1f96d1cbed815d92582863c424af" gracePeriod=45 Nov 03 01:18:52.284006 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.283976 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 01:18:52.312468 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.312446 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 01:18:52.327723 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.327693 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-tmp-dir\") pod \"f60672ea-e415-4096-bf4f-4689d4194dbf\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " Nov 03 01:18:52.327875 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.327771 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf4988a-ac9f-446e-91ec-28ee63f944a7-kube-api-access-b662r\" (UniqueName: \"kubernetes.io/projected/eaf4988a-ac9f-446e-91ec-28ee63f944a7-kube-api-access-b662r\") pod \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " Nov 03 01:18:52.327875 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.327842 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-model-cache\") pod \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " Nov 03 01:18:52.327875 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.327867 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-model-cache\") pod \"f60672ea-e415-4096-bf4f-4689d4194dbf\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " Nov 03 01:18:52.328044 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.327884 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tmp-dir\") pod \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " Nov 03 01:18:52.328044 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.327916 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f60672ea-e415-4096-bf4f-4689d4194dbf-tls-certs\" (UniqueName: \"kubernetes.io/secret/f60672ea-e415-4096-bf4f-4689d4194dbf-tls-certs\") pod \"f60672ea-e415-4096-bf4f-4689d4194dbf\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " Nov 03 01:18:52.328044 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.327935 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-home\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-home\") pod \"f60672ea-e415-4096-bf4f-4689d4194dbf\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " Nov 03 01:18:52.328044 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.327959 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-home\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-home\") pod \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " Nov 03 01:18:52.328044 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328028 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-dshm\") pod \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " Nov 03 01:18:52.328318 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328061 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-dshm\") pod \"f60672ea-e415-4096-bf4f-4689d4194dbf\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " Nov 03 01:18:52.328318 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328085 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tls-certs\") pod \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\" (UID: \"eaf4988a-ac9f-446e-91ec-28ee63f944a7\") " Nov 03 01:18:52.328318 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328141 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f60672ea-e415-4096-bf4f-4689d4194dbf-kube-api-access-j5bdj\" (UniqueName: \"kubernetes.io/projected/f60672ea-e415-4096-bf4f-4689d4194dbf-kube-api-access-j5bdj\") pod \"f60672ea-e415-4096-bf4f-4689d4194dbf\" (UID: \"f60672ea-e415-4096-bf4f-4689d4194dbf\") " Nov 03 01:18:52.328318 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328157 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-model-cache" pod "f60672ea-e415-4096-bf4f-4689d4194dbf" (UID: "f60672ea-e415-4096-bf4f-4689d4194dbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.328318 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328158 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tmp-dir" pod "eaf4988a-ac9f-446e-91ec-28ee63f944a7" (UID: "eaf4988a-ac9f-446e-91ec-28ee63f944a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.328318 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328179 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-model-cache" pod "eaf4988a-ac9f-446e-91ec-28ee63f944a7" (UID: "eaf4988a-ac9f-446e-91ec-28ee63f944a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.328318 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328198 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-home" pod "f60672ea-e415-4096-bf4f-4689d4194dbf" (UID: "f60672ea-e415-4096-bf4f-4689d4194dbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.328667 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328405 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-home" pod "eaf4988a-ac9f-446e-91ec-28ee63f944a7" (UID: "eaf4988a-ac9f-446e-91ec-28ee63f944a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.328667 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328613 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.328667 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328632 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.328855 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328646 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.328855 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328702 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.328855 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328727 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.328855 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.328699 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-tmp-dir" pod "f60672ea-e415-4096-bf4f-4689d4194dbf" (UID: "f60672ea-e415-4096-bf4f-4689d4194dbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.330944 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.330908 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf4988a-ac9f-446e-91ec-28ee63f944a7-kube-api-access-b662r" pod "eaf4988a-ac9f-446e-91ec-28ee63f944a7" (UID: "eaf4988a-ac9f-446e-91ec-28ee63f944a7"). InnerVolumeSpecName "kube-api-access-b662r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:52.331063 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.330994 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-dshm" pod "eaf4988a-ac9f-446e-91ec-28ee63f944a7" (UID: "eaf4988a-ac9f-446e-91ec-28ee63f944a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.331129 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.331072 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f60672ea-e415-4096-bf4f-4689d4194dbf-tls-certs" pod "f60672ea-e415-4096-bf4f-4689d4194dbf" (UID: "f60672ea-e415-4096-bf4f-4689d4194dbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:52.331129 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.331076 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-dshm" pod "f60672ea-e415-4096-bf4f-4689d4194dbf" (UID: "f60672ea-e415-4096-bf4f-4689d4194dbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.331227 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.331162 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f60672ea-e415-4096-bf4f-4689d4194dbf-kube-api-access-j5bdj" pod "f60672ea-e415-4096-bf4f-4689d4194dbf" (UID: "f60672ea-e415-4096-bf4f-4689d4194dbf"). InnerVolumeSpecName "kube-api-access-j5bdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:52.331459 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.331443 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tls-certs" pod "eaf4988a-ac9f-446e-91ec-28ee63f944a7" (UID: "eaf4988a-ac9f-446e-91ec-28ee63f944a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:52.429618 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.429585 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f60672ea-e415-4096-bf4f-4689d4194dbf-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.429618 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.429616 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaf4988a-ac9f-446e-91ec-28ee63f944a7-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.429618 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.429626 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.429869 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.429634 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf4988a-ac9f-446e-91ec-28ee63f944a7-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.429869 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.429644 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5bdj\" (UniqueName: \"kubernetes.io/projected/f60672ea-e415-4096-bf4f-4689d4194dbf-kube-api-access-j5bdj\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.429869 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.429652 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60672ea-e415-4096-bf4f-4689d4194dbf-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.429869 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.429661 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b662r\" (UniqueName: \"kubernetes.io/projected/eaf4988a-ac9f-446e-91ec-28ee63f944a7-kube-api-access-b662r\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.894056 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.894017 6400 generic.go:356] "Generic (PLEG): container finished" podID="f60672ea-e415-4096-bf4f-4689d4194dbf" containerID="4bb68f545c59da36c217afa86f128b14d7fa1f96d1cbed815d92582863c424af" exitCode=0 Nov 03 01:18:52.894255 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.894065 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" Nov 03 01:18:52.894255 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.894109 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" event={"ID":"f60672ea-e415-4096-bf4f-4689d4194dbf","Type":"ContainerDied","Data":"4bb68f545c59da36c217afa86f128b14d7fa1f96d1cbed815d92582863c424af"} Nov 03 01:18:52.894255 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.894157 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr" event={"ID":"f60672ea-e415-4096-bf4f-4689d4194dbf","Type":"ContainerDied","Data":"d4e1bcb36c3afb8a2b68f75b2e673055b092b5f5e05d3fee88358911839ac8a9"} Nov 03 01:18:52.894255 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.894175 6400 scope.go:122] "RemoveContainer" containerID="4bb68f545c59da36c217afa86f128b14d7fa1f96d1cbed815d92582863c424af" Nov 03 01:18:52.895731 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.895706 6400 generic.go:356] "Generic (PLEG): container finished" podID="eaf4988a-ac9f-446e-91ec-28ee63f944a7" containerID="03e0dd81c9b62cca3890c26b7f8cf03aa607b5d7b936c6aee7aa6b245fb43050" exitCode=0 Nov 03 01:18:52.895863 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.895760 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" Nov 03 01:18:52.895863 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.895794 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" event={"ID":"eaf4988a-ac9f-446e-91ec-28ee63f944a7","Type":"ContainerDied","Data":"03e0dd81c9b62cca3890c26b7f8cf03aa607b5d7b936c6aee7aa6b245fb43050"} Nov 03 01:18:52.895962 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.895865 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5" event={"ID":"eaf4988a-ac9f-446e-91ec-28ee63f944a7","Type":"ContainerDied","Data":"e6e23c30d924a9cde90b52ad7ddbbd9c46e7cfc5283dcb06fbcc424c0fd319ea"} Nov 03 01:18:52.903545 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.903518 6400 scope.go:122] "RemoveContainer" containerID="4bb68f545c59da36c217afa86f128b14d7fa1f96d1cbed815d92582863c424af" Nov 03 01:18:52.903882 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:18:52.903859 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb68f545c59da36c217afa86f128b14d7fa1f96d1cbed815d92582863c424af\": container with ID starting with 4bb68f545c59da36c217afa86f128b14d7fa1f96d1cbed815d92582863c424af not found: ID does not exist" containerID="4bb68f545c59da36c217afa86f128b14d7fa1f96d1cbed815d92582863c424af" Nov 03 01:18:52.903958 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.903890 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb68f545c59da36c217afa86f128b14d7fa1f96d1cbed815d92582863c424af"} err="failed to get container status \"4bb68f545c59da36c217afa86f128b14d7fa1f96d1cbed815d92582863c424af\": rpc error: code = NotFound desc = could not find container \"4bb68f545c59da36c217afa86f128b14d7fa1f96d1cbed815d92582863c424af\": container with ID starting with 4bb68f545c59da36c217afa86f128b14d7fa1f96d1cbed815d92582863c424af not found: ID does not exist" Nov 03 01:18:52.903958 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.903906 6400 scope.go:122] "RemoveContainer" containerID="03e0dd81c9b62cca3890c26b7f8cf03aa607b5d7b936c6aee7aa6b245fb43050" Nov 03 01:18:52.912442 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.912424 6400 scope.go:122] "RemoveContainer" containerID="03e0dd81c9b62cca3890c26b7f8cf03aa607b5d7b936c6aee7aa6b245fb43050" Nov 03 01:18:52.912739 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:18:52.912710 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e0dd81c9b62cca3890c26b7f8cf03aa607b5d7b936c6aee7aa6b245fb43050\": container with ID starting with 03e0dd81c9b62cca3890c26b7f8cf03aa607b5d7b936c6aee7aa6b245fb43050 not found: ID does not exist" containerID="03e0dd81c9b62cca3890c26b7f8cf03aa607b5d7b936c6aee7aa6b245fb43050" Nov 03 01:18:52.912800 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:52.912747 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e0dd81c9b62cca3890c26b7f8cf03aa607b5d7b936c6aee7aa6b245fb43050"} err="failed to get container status \"03e0dd81c9b62cca3890c26b7f8cf03aa607b5d7b936c6aee7aa6b245fb43050\": rpc error: code = NotFound desc = could not find container \"03e0dd81c9b62cca3890c26b7f8cf03aa607b5d7b936c6aee7aa6b245fb43050\": container with ID starting with 03e0dd81c9b62cca3890c26b7f8cf03aa607b5d7b936c6aee7aa6b245fb43050 not found: ID does not exist" Nov 03 01:18:53.060508 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:53.060476 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5"] Nov 03 01:18:53.063839 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:53.063795 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/distinct-2-w3-d7bfc2-kserve-67bdb5d8b5-674z5"] Nov 03 01:18:53.180406 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:53.180315 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf4988a-ac9f-446e-91ec-28ee63f944a7" path="/var/lib/kubelet/pods/eaf4988a-ac9f-446e-91ec-28ee63f944a7/volumes" Nov 03 01:18:53.180711 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:53.180691 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr"] Nov 03 01:18:53.180841 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:53.180734 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/distinct-w3-d7bfc2-kserve-84d6f4888b-tmlpr"] Nov 03 01:18:55.178548 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:55.178509 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f60672ea-e415-4096-bf4f-4689d4194dbf" path="/var/lib/kubelet/pods/f60672ea-e415-4096-bf4f-4689d4194dbf/volumes" Nov 03 01:18:55.179836 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:55.179795 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:55.353508 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:55.353465 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:55.508216 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:55.508137 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:56.610244 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:56.610205 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:57.000115 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:18:57.000028 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:57.151373 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:18:57.151342 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:19:00.890159 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:00.890127 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:19:01.326697 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:01.326661 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:01.815961 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:01.815929 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:06.622934 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:06.622894 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:11.315312 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:11.315276 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:14.729073 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:14.729035 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:14.903473 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:14.903433 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:15.063765 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:15.063737 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:16.032330 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:16.032296 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:19.990152 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:19.990123 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr"] Nov 03 01:19:19.990625 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:19.990328 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" podUID="bc8bfcd4-3359-4a5f-aeef-50516cf73fa3" containerName="maas-api" containerID="cri-o://f507790e8e9a2c933e001e261c45ec5ba7c707981cc3d478b81d1f71fa7587b1" gracePeriod=30 Nov 03 01:19:20.122282 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.122245 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:20.295146 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.295118 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 01:19:20.443105 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.443080 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-maas-api-tls\") pod \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " Nov 03 01:19:20.443299 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.443114 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-openshift-service-ca\") pod \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " Nov 03 01:19:20.443299 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.443143 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-metrics-certs\") pod \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " Nov 03 01:19:20.443299 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.443270 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-kube-api-access-5p4fg\" (UniqueName: \"kubernetes.io/projected/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-kube-api-access-5p4fg\") pod \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\" (UID: \"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3\") " Nov 03 01:19:20.443555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.443522 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-openshift-service-ca" pod "bc8bfcd4-3359-4a5f-aeef-50516cf73fa3" (UID: "bc8bfcd4-3359-4a5f-aeef-50516cf73fa3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:20.443776 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.443753 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:20.445680 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.445653 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-metrics-certs" pod "bc8bfcd4-3359-4a5f-aeef-50516cf73fa3" (UID: "bc8bfcd4-3359-4a5f-aeef-50516cf73fa3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:19:20.445779 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.445725 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-maas-api-tls" pod "bc8bfcd4-3359-4a5f-aeef-50516cf73fa3" (UID: "bc8bfcd4-3359-4a5f-aeef-50516cf73fa3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:19:20.445940 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.445785 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-kube-api-access-5p4fg" pod "bc8bfcd4-3359-4a5f-aeef-50516cf73fa3" (UID: "bc8bfcd4-3359-4a5f-aeef-50516cf73fa3"). InnerVolumeSpecName "kube-api-access-5p4fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:20.545035 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.544998 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:20.545035 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.545036 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:20.545313 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:20.545051 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p4fg\" (UniqueName: \"kubernetes.io/projected/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3-kube-api-access-5p4fg\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:21.065487 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:21.065448 6400 generic.go:356] "Generic (PLEG): container finished" podID="bc8bfcd4-3359-4a5f-aeef-50516cf73fa3" containerID="f507790e8e9a2c933e001e261c45ec5ba7c707981cc3d478b81d1f71fa7587b1" exitCode=0 Nov 03 01:19:21.066017 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:21.065533 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" event={"ID":"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3","Type":"ContainerDied","Data":"f507790e8e9a2c933e001e261c45ec5ba7c707981cc3d478b81d1f71fa7587b1"} Nov 03 01:19:21.066017 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:21.065551 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" Nov 03 01:19:21.066017 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:21.065592 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr" event={"ID":"bc8bfcd4-3359-4a5f-aeef-50516cf73fa3","Type":"ContainerDied","Data":"00953047bb1ce9600072303b4844acc62ff5fabfdf2ff2858505309a2274c938"} Nov 03 01:19:21.066017 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:21.065616 6400 scope.go:122] "RemoveContainer" containerID="f507790e8e9a2c933e001e261c45ec5ba7c707981cc3d478b81d1f71fa7587b1" Nov 03 01:19:21.075744 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:21.075724 6400 scope.go:122] "RemoveContainer" containerID="f507790e8e9a2c933e001e261c45ec5ba7c707981cc3d478b81d1f71fa7587b1" Nov 03 01:19:21.076087 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:19:21.076062 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f507790e8e9a2c933e001e261c45ec5ba7c707981cc3d478b81d1f71fa7587b1\": container with ID starting with f507790e8e9a2c933e001e261c45ec5ba7c707981cc3d478b81d1f71fa7587b1 not found: ID does not exist" containerID="f507790e8e9a2c933e001e261c45ec5ba7c707981cc3d478b81d1f71fa7587b1" Nov 03 01:19:21.076169 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:21.076107 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f507790e8e9a2c933e001e261c45ec5ba7c707981cc3d478b81d1f71fa7587b1"} err="failed to get container status \"f507790e8e9a2c933e001e261c45ec5ba7c707981cc3d478b81d1f71fa7587b1\": rpc error: code = NotFound desc = could not find container \"f507790e8e9a2c933e001e261c45ec5ba7c707981cc3d478b81d1f71fa7587b1\": container with ID starting with f507790e8e9a2c933e001e261c45ec5ba7c707981cc3d478b81d1f71fa7587b1 not found: ID does not exist" Nov 03 01:19:21.176984 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:21.176951 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:21.228030 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:21.227991 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr"] Nov 03 01:19:21.276338 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:21.276307 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7bfc2-c4c64df48-qntvr"] Nov 03 01:19:21.338995 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:21.338909 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:21.920038 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:21.920004 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:23.178224 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:23.178189 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc8bfcd4-3359-4a5f-aeef-50516cf73fa3" path="/var/lib/kubelet/pods/bc8bfcd4-3359-4a5f-aeef-50516cf73fa3/volumes" Nov 03 01:19:23.653166 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:23.653134 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:26.643017 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:26.642974 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:27.152364 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:19:27.152319 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:19:28.789948 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:28.789920 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:29.865381 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:29.865350 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:30.333947 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:30.333911 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:31.584296 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:31.584181 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:35.629170 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:35.629139 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz"] Nov 03 01:19:35.629742 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:35.629335 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" podUID="29761a2e-1646-4996-9f3e-89a0ace1588d" containerName="maas-api" containerID="cri-o://e3354647da10f814071d995e89732b14d7f02acd6c4861ebd5f8ea2f2e159788" gracePeriod=30 Nov 03 01:19:35.758609 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:35.758580 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:35.929049 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:35.929018 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 01:19:36.021268 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.021145 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-metrics-certs\") pod \"29761a2e-1646-4996-9f3e-89a0ace1588d\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " Nov 03 01:19:36.021268 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.021214 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29761a2e-1646-4996-9f3e-89a0ace1588d-kube-api-access-q4wbg\" (UniqueName: \"kubernetes.io/projected/29761a2e-1646-4996-9f3e-89a0ace1588d-kube-api-access-q4wbg\") pod \"29761a2e-1646-4996-9f3e-89a0ace1588d\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " Nov 03 01:19:36.021268 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.021258 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29761a2e-1646-4996-9f3e-89a0ace1588d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29761a2e-1646-4996-9f3e-89a0ace1588d-openshift-service-ca\") pod \"29761a2e-1646-4996-9f3e-89a0ace1588d\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " Nov 03 01:19:36.021711 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.021407 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-maas-api-tls\") pod \"29761a2e-1646-4996-9f3e-89a0ace1588d\" (UID: \"29761a2e-1646-4996-9f3e-89a0ace1588d\") " Nov 03 01:19:36.024487 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.021975 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29761a2e-1646-4996-9f3e-89a0ace1588d-openshift-service-ca" pod "29761a2e-1646-4996-9f3e-89a0ace1588d" (UID: "29761a2e-1646-4996-9f3e-89a0ace1588d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:36.025327 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.025271 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-metrics-certs" pod "29761a2e-1646-4996-9f3e-89a0ace1588d" (UID: "29761a2e-1646-4996-9f3e-89a0ace1588d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:19:36.025327 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.025287 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29761a2e-1646-4996-9f3e-89a0ace1588d-kube-api-access-q4wbg" pod "29761a2e-1646-4996-9f3e-89a0ace1588d" (UID: "29761a2e-1646-4996-9f3e-89a0ace1588d"). InnerVolumeSpecName "kube-api-access-q4wbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:36.028840 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.027763 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-maas-api-tls" pod "29761a2e-1646-4996-9f3e-89a0ace1588d" (UID: "29761a2e-1646-4996-9f3e-89a0ace1588d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:19:36.123341 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.123309 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.123341 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.123342 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29761a2e-1646-4996-9f3e-89a0ace1588d-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.123551 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.123358 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4wbg\" (UniqueName: \"kubernetes.io/projected/29761a2e-1646-4996-9f3e-89a0ace1588d-kube-api-access-q4wbg\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.123551 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.123373 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29761a2e-1646-4996-9f3e-89a0ace1588d-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.142255 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.142218 6400 generic.go:356] "Generic (PLEG): container finished" podID="29761a2e-1646-4996-9f3e-89a0ace1588d" containerID="e3354647da10f814071d995e89732b14d7f02acd6c4861ebd5f8ea2f2e159788" exitCode=0 Nov 03 01:19:36.142386 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.142264 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" Nov 03 01:19:36.142386 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.142305 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" event={"ID":"29761a2e-1646-4996-9f3e-89a0ace1588d","Type":"ContainerDied","Data":"e3354647da10f814071d995e89732b14d7f02acd6c4861ebd5f8ea2f2e159788"} Nov 03 01:19:36.142386 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.142363 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz" event={"ID":"29761a2e-1646-4996-9f3e-89a0ace1588d","Type":"ContainerDied","Data":"5b23848bbbfdcdb8995e53c24a55328ed027142c0be22378965d3b360f68758d"} Nov 03 01:19:36.142501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.142392 6400 scope.go:122] "RemoveContainer" containerID="e3354647da10f814071d995e89732b14d7f02acd6c4861ebd5f8ea2f2e159788" Nov 03 01:19:36.159635 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.159614 6400 scope.go:122] "RemoveContainer" containerID="e3354647da10f814071d995e89732b14d7f02acd6c4861ebd5f8ea2f2e159788" Nov 03 01:19:36.159938 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:19:36.159918 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3354647da10f814071d995e89732b14d7f02acd6c4861ebd5f8ea2f2e159788\": container with ID starting with e3354647da10f814071d995e89732b14d7f02acd6c4861ebd5f8ea2f2e159788 not found: ID does not exist" containerID="e3354647da10f814071d995e89732b14d7f02acd6c4861ebd5f8ea2f2e159788" Nov 03 01:19:36.160004 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.159947 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3354647da10f814071d995e89732b14d7f02acd6c4861ebd5f8ea2f2e159788"} err="failed to get container status \"e3354647da10f814071d995e89732b14d7f02acd6c4861ebd5f8ea2f2e159788\": rpc error: code = NotFound desc = could not find container \"e3354647da10f814071d995e89732b14d7f02acd6c4861ebd5f8ea2f2e159788\": container with ID starting with e3354647da10f814071d995e89732b14d7f02acd6c4861ebd5f8ea2f2e159788 not found: ID does not exist" Nov 03 01:19:36.306380 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.306338 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz"] Nov 03 01:19:36.309843 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.309804 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-328e1b-7d59dd6548-kjtnz"] Nov 03 01:19:36.364493 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:36.364467 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:19:37.177878 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:37.177815 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29761a2e-1646-4996-9f3e-89a0ace1588d" path="/var/lib/kubelet/pods/29761a2e-1646-4996-9f3e-89a0ace1588d/volumes" Nov 03 01:19:37.536290 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:37.536253 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:42.212942 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:42.212906 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:44.403321 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:44.403284 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:44.613314 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:44.613282 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:45.908150 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:45.908105 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:46.386581 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:46.386546 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:46.859068 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:46.859036 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:47.104869 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:47.104771 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:50.012190 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:50.012152 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:50.234299 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:50.234264 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:51.792272 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:51.792238 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs"] Nov 03 01:19:51.911445 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:51.911408 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:52.107734 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:52.107650 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:54.701438 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:54.701342 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:56.606528 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:56.606501 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:57.152292 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:19:57.152258 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:19:58.113275 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:19:58.113241 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:01.327539 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:01.327506 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:03.911353 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:03.911314 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:04.076841 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:04.076691 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:04.951649 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:04.951614 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:05.228220 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:05.228143 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:05.690348 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:05.690313 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:06.235457 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:06.235421 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:06.813892 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:06.813807 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" podUID="dfbe3d21-ff1e-4d7e-8e16-705478b6f95d" containerName="main" containerID="cri-o://08761ea40ed934e9a1091ed36c7711bc77e932e044db9cf9fd88192d17e3a99b" gracePeriod=45 Nov 03 01:20:07.069646 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.069563 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 01:20:07.148662 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.148633 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-home\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-home\") pod \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " Nov 03 01:20:07.148807 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.148685 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-kube-api-access-kwfwn\" (UniqueName: \"kubernetes.io/projected/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-kube-api-access-kwfwn\") pod \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " Nov 03 01:20:07.148807 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.148722 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-dshm\") pod \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " Nov 03 01:20:07.148807 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.148785 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tls-certs\") pod \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " Nov 03 01:20:07.149011 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.148891 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-model-cache\") pod \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " Nov 03 01:20:07.149011 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.148921 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tmp-dir\") pod \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\" (UID: \"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d\") " Nov 03 01:20:07.149011 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.148935 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-home" pod "dfbe3d21-ff1e-4d7e-8e16-705478b6f95d" (UID: "dfbe3d21-ff1e-4d7e-8e16-705478b6f95d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.149169 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.149148 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-model-cache" pod "dfbe3d21-ff1e-4d7e-8e16-705478b6f95d" (UID: "dfbe3d21-ff1e-4d7e-8e16-705478b6f95d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.149213 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.149168 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tmp-dir" pod "dfbe3d21-ff1e-4d7e-8e16-705478b6f95d" (UID: "dfbe3d21-ff1e-4d7e-8e16-705478b6f95d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.149480 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.149432 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.149480 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.149471 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.149480 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.149482 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.151104 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.151077 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tls-certs" pod "dfbe3d21-ff1e-4d7e-8e16-705478b6f95d" (UID: "dfbe3d21-ff1e-4d7e-8e16-705478b6f95d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:07.151209 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.151121 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-dshm" pod "dfbe3d21-ff1e-4d7e-8e16-705478b6f95d" (UID: "dfbe3d21-ff1e-4d7e-8e16-705478b6f95d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.151209 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.151127 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-kube-api-access-kwfwn" pod "dfbe3d21-ff1e-4d7e-8e16-705478b6f95d" (UID: "dfbe3d21-ff1e-4d7e-8e16-705478b6f95d"). InnerVolumeSpecName "kube-api-access-kwfwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:07.250795 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.250762 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwfwn\" (UniqueName: \"kubernetes.io/projected/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-kube-api-access-kwfwn\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.251260 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.251236 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.251260 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.251260 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.311986 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.311954 6400 generic.go:356] "Generic (PLEG): container finished" podID="dfbe3d21-ff1e-4d7e-8e16-705478b6f95d" containerID="08761ea40ed934e9a1091ed36c7711bc77e932e044db9cf9fd88192d17e3a99b" exitCode=0 Nov 03 01:20:07.312132 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.311997 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" Nov 03 01:20:07.312132 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.312030 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" event={"ID":"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d","Type":"ContainerDied","Data":"08761ea40ed934e9a1091ed36c7711bc77e932e044db9cf9fd88192d17e3a99b"} Nov 03 01:20:07.312132 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.312066 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs" event={"ID":"dfbe3d21-ff1e-4d7e-8e16-705478b6f95d","Type":"ContainerDied","Data":"09a5ad586bb4d0f425d3ebc736d0cb2b2a08e4be93d31fc0e503b18e1547129e"} Nov 03 01:20:07.312132 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.312085 6400 scope.go:122] "RemoveContainer" containerID="08761ea40ed934e9a1091ed36c7711bc77e932e044db9cf9fd88192d17e3a99b" Nov 03 01:20:07.320383 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.320366 6400 scope.go:122] "RemoveContainer" containerID="08761ea40ed934e9a1091ed36c7711bc77e932e044db9cf9fd88192d17e3a99b" Nov 03 01:20:07.320647 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:20:07.320624 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08761ea40ed934e9a1091ed36c7711bc77e932e044db9cf9fd88192d17e3a99b\": container with ID starting with 08761ea40ed934e9a1091ed36c7711bc77e932e044db9cf9fd88192d17e3a99b not found: ID does not exist" containerID="08761ea40ed934e9a1091ed36c7711bc77e932e044db9cf9fd88192d17e3a99b" Nov 03 01:20:07.320698 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.320658 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08761ea40ed934e9a1091ed36c7711bc77e932e044db9cf9fd88192d17e3a99b"} err="failed to get container status \"08761ea40ed934e9a1091ed36c7711bc77e932e044db9cf9fd88192d17e3a99b\": rpc error: code = NotFound desc = could not find container \"08761ea40ed934e9a1091ed36c7711bc77e932e044db9cf9fd88192d17e3a99b\": container with ID starting with 08761ea40ed934e9a1091ed36c7711bc77e932e044db9cf9fd88192d17e3a99b not found: ID does not exist" Nov 03 01:20:07.475440 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.475027 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs"] Nov 03 01:20:07.476487 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:07.476462 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/distinct-2-w2-531ee4-kserve-5ff474499-s4ghs"] Nov 03 01:20:09.178780 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:09.178743 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfbe3d21-ff1e-4d7e-8e16-705478b6f95d" path="/var/lib/kubelet/pods/dfbe3d21-ff1e-4d7e-8e16-705478b6f95d/volumes" Nov 03 01:20:10.575699 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:10.575670 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:11.039115 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:11.039069 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:11.206630 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:11.206591 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:11.364109 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:11.364028 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:12.141573 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:12.141536 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:14.102060 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:14.102022 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:16.003333 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:16.003302 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:17.592865 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:17.592801 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:19.188687 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:19.188652 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:20:20.796259 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:20.796228 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:24.021105 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:24.021056 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:24.912522 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:24.912478 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:25.499499 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:25.499458 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:26.685740 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:26.685708 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:27.153526 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:20:27.153489 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:20:28.869646 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:28.869610 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:29.078490 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:29.078457 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:29.697318 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:29.697279 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:30.013308 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:30.013211 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:30.178465 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:30.178429 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:31.401942 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:31.401904 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:32.528650 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:32.528615 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:32.700702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:32.700670 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:34.015314 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:34.015282 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:34.537389 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:34.537348 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:34.704415 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:34.704379 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:35.358741 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:35.358688 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:36.266656 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:36.266617 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:36.419513 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:36.419482 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:41.112273 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:41.112240 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:42.141696 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:42.141659 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:43.385256 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:43.385224 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:43.585121 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:43.585082 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:43.973733 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:43.973691 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h"] Nov 03 01:20:44.002716 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:44.002681 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:46.209021 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:46.208991 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:47.801035 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:47.801000 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:50.456710 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:50.456677 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:51.083289 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:51.083237 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:51.238144 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:51.238109 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:52.227871 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:52.227811 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:56.584935 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:56.584901 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:20:56.932120 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:56.932035 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:57.153462 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:20:57.153423 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:20:58.995254 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:58.995193 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" podUID="03e891a7-9472-4300-b9f0-b75a745280b9" containerName="main" containerID="cri-o://cc0a340e21429cb293e76412e99300a393466d7d521eb0bf9d97e942eb2baf8d" gracePeriod=45 Nov 03 01:20:59.104013 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.103973 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:59.276965 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.276939 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:20:59.297092 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.297059 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:59.355335 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.355295 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-tmp-dir\") pod \"03e891a7-9472-4300-b9f0-b75a745280b9\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " Nov 03 01:20:59.355537 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.355359 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e891a7-9472-4300-b9f0-b75a745280b9-kube-api-access-hbhsn\" (UniqueName: \"kubernetes.io/projected/03e891a7-9472-4300-b9f0-b75a745280b9-kube-api-access-hbhsn\") pod \"03e891a7-9472-4300-b9f0-b75a745280b9\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " Nov 03 01:20:59.355537 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.355397 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-home\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-home\") pod \"03e891a7-9472-4300-b9f0-b75a745280b9\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " Nov 03 01:20:59.355537 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.355447 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-dshm\") pod \"03e891a7-9472-4300-b9f0-b75a745280b9\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " Nov 03 01:20:59.355710 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.355543 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03e891a7-9472-4300-b9f0-b75a745280b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/03e891a7-9472-4300-b9f0-b75a745280b9-tls-certs\") pod \"03e891a7-9472-4300-b9f0-b75a745280b9\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " Nov 03 01:20:59.355710 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.355574 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-model-cache\") pod \"03e891a7-9472-4300-b9f0-b75a745280b9\" (UID: \"03e891a7-9472-4300-b9f0-b75a745280b9\") " Nov 03 01:20:59.355710 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.355628 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-tmp-dir" pod "03e891a7-9472-4300-b9f0-b75a745280b9" (UID: "03e891a7-9472-4300-b9f0-b75a745280b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:59.355710 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.355661 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-home" pod "03e891a7-9472-4300-b9f0-b75a745280b9" (UID: "03e891a7-9472-4300-b9f0-b75a745280b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:59.355978 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.355958 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:59.356042 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.355984 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:59.356042 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.355955 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-model-cache" pod "03e891a7-9472-4300-b9f0-b75a745280b9" (UID: "03e891a7-9472-4300-b9f0-b75a745280b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:59.357856 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.357806 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-dshm" pod "03e891a7-9472-4300-b9f0-b75a745280b9" (UID: "03e891a7-9472-4300-b9f0-b75a745280b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:59.358027 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.358012 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e891a7-9472-4300-b9f0-b75a745280b9-tls-certs" pod "03e891a7-9472-4300-b9f0-b75a745280b9" (UID: "03e891a7-9472-4300-b9f0-b75a745280b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:59.358117 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.358097 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e891a7-9472-4300-b9f0-b75a745280b9-kube-api-access-hbhsn" pod "03e891a7-9472-4300-b9f0-b75a745280b9" (UID: "03e891a7-9472-4300-b9f0-b75a745280b9"). InnerVolumeSpecName "kube-api-access-hbhsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:59.457257 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.457215 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e891a7-9472-4300-b9f0-b75a745280b9-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:59.457257 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.457258 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:59.457490 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.457275 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbhsn\" (UniqueName: \"kubernetes.io/projected/03e891a7-9472-4300-b9f0-b75a745280b9-kube-api-access-hbhsn\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:59.457490 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.457290 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e891a7-9472-4300-b9f0-b75a745280b9-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:59.645959 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.645923 6400 generic.go:356] "Generic (PLEG): container finished" podID="03e891a7-9472-4300-b9f0-b75a745280b9" containerID="cc0a340e21429cb293e76412e99300a393466d7d521eb0bf9d97e942eb2baf8d" exitCode=0 Nov 03 01:20:59.646178 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.645979 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" Nov 03 01:20:59.646178 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.646018 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" event={"ID":"03e891a7-9472-4300-b9f0-b75a745280b9","Type":"ContainerDied","Data":"cc0a340e21429cb293e76412e99300a393466d7d521eb0bf9d97e942eb2baf8d"} Nov 03 01:20:59.646178 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.646072 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h" event={"ID":"03e891a7-9472-4300-b9f0-b75a745280b9","Type":"ContainerDied","Data":"ffe5dfc1033c96f1884c5bc37d6b9647e4fe181bd5fefdaead85d39096b5ccf2"} Nov 03 01:20:59.646178 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.646100 6400 scope.go:122] "RemoveContainer" containerID="cc0a340e21429cb293e76412e99300a393466d7d521eb0bf9d97e942eb2baf8d" Nov 03 01:20:59.655739 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.655718 6400 scope.go:122] "RemoveContainer" containerID="cc0a340e21429cb293e76412e99300a393466d7d521eb0bf9d97e942eb2baf8d" Nov 03 01:20:59.656063 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:20:59.656044 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc0a340e21429cb293e76412e99300a393466d7d521eb0bf9d97e942eb2baf8d\": container with ID starting with cc0a340e21429cb293e76412e99300a393466d7d521eb0bf9d97e942eb2baf8d not found: ID does not exist" containerID="cc0a340e21429cb293e76412e99300a393466d7d521eb0bf9d97e942eb2baf8d" Nov 03 01:20:59.656141 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.656075 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0a340e21429cb293e76412e99300a393466d7d521eb0bf9d97e942eb2baf8d"} err="failed to get container status \"cc0a340e21429cb293e76412e99300a393466d7d521eb0bf9d97e942eb2baf8d\": rpc error: code = NotFound desc = could not find container \"cc0a340e21429cb293e76412e99300a393466d7d521eb0bf9d97e942eb2baf8d\": container with ID starting with cc0a340e21429cb293e76412e99300a393466d7d521eb0bf9d97e942eb2baf8d not found: ID does not exist" Nov 03 01:20:59.810211 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.810180 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h"] Nov 03 01:20:59.859253 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:20:59.859215 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b2e795/sub-test-model-b2e795-kserve-6758474ccc-5kl2h"] Nov 03 01:21:00.553508 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:00.553469 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:01.044302 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:01.044257 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:01.180572 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:01.180538 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e891a7-9472-4300-b9f0-b75a745280b9" path="/var/lib/kubelet/pods/03e891a7-9472-4300-b9f0-b75a745280b9/volumes" Nov 03 01:21:01.216657 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:01.216611 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:03.164718 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:03.164679 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:04.187055 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:04.187012 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59"] Nov 03 01:21:04.224391 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:04.224348 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:04.673006 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:04.672966 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:07.818040 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:07.817986 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:10.012521 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:10.012488 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:21:11.029939 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:11.029904 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:11.396986 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:11.396945 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:11.572094 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:11.572051 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:12.552387 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:12.552354 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:12.701192 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:12.701122 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:12.921291 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:12.921197 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:13.357366 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:13.357337 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk"] Nov 03 01:21:13.357598 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:13.357577 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" podUID="728c90f9-2248-493e-adc6-7e51e412154d" containerName="istio-proxy" containerID="cri-o://4873956bad3c1b5220f2183e23dcdb004bd4e97b9c52b756a478ad1f7e9f8132" gracePeriod=30 Nov 03 01:21:13.381910 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:13.381876 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:13.832624 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:13.832587 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:16.532109 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:16.532074 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:17.700930 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:17.700878 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:18.618899 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.618869 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 01:21:18.771111 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.771021 6400 generic.go:356] "Generic (PLEG): container finished" podID="728c90f9-2248-493e-adc6-7e51e412154d" containerID="4873956bad3c1b5220f2183e23dcdb004bd4e97b9c52b756a478ad1f7e9f8132" exitCode=0 Nov 03 01:21:18.771111 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.771055 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" event={"ID":"728c90f9-2248-493e-adc6-7e51e412154d","Type":"ContainerDied","Data":"4873956bad3c1b5220f2183e23dcdb004bd4e97b9c52b756a478ad1f7e9f8132"} Nov 03 01:21:18.771111 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.771076 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" Nov 03 01:21:18.771670 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.771113 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk" event={"ID":"728c90f9-2248-493e-adc6-7e51e412154d","Type":"ContainerDied","Data":"a4365e80d15f312c90120e66f9b527fa970a46b27376ac5b1e8887e9209ec36c"} Nov 03 01:21:18.771670 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.771142 6400 scope.go:122] "RemoveContainer" containerID="4873956bad3c1b5220f2183e23dcdb004bd4e97b9c52b756a478ad1f7e9f8132" Nov 03 01:21:18.781259 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.781237 6400 scope.go:122] "RemoveContainer" containerID="67935ffc9efc96bfe0d96a47b96cc311f1223647cb8edc5bb3f2b40d03a9dca3" Nov 03 01:21:18.783586 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.783559 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/728c90f9-2248-493e-adc6-7e51e412154d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728c90f9-2248-493e-adc6-7e51e412154d-istiod-ca-cert\") pod \"728c90f9-2248-493e-adc6-7e51e412154d\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " Nov 03 01:21:18.783728 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.783621 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-certs\") pod \"728c90f9-2248-493e-adc6-7e51e412154d\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " Nov 03 01:21:18.783728 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.783647 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-socket\") pod \"728c90f9-2248-493e-adc6-7e51e412154d\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " Nov 03 01:21:18.783728 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.783687 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-data\") pod \"728c90f9-2248-493e-adc6-7e51e412154d\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " Nov 03 01:21:18.783948 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.783732 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/728c90f9-2248-493e-adc6-7e51e412154d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728c90f9-2248-493e-adc6-7e51e412154d-istio-podinfo\") pod \"728c90f9-2248-493e-adc6-7e51e412154d\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " Nov 03 01:21:18.783948 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.783856 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-envoy\") pod \"728c90f9-2248-493e-adc6-7e51e412154d\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " Nov 03 01:21:18.784052 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.783968 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-credential-socket\") pod \"728c90f9-2248-493e-adc6-7e51e412154d\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " Nov 03 01:21:18.784104 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.784064 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-istio-token\" (UniqueName: \"kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-istio-token\") pod \"728c90f9-2248-493e-adc6-7e51e412154d\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " Nov 03 01:21:18.784104 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.784094 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-kube-api-access-p62zw\" (UniqueName: \"kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-kube-api-access-p62zw\") pod \"728c90f9-2248-493e-adc6-7e51e412154d\" (UID: \"728c90f9-2248-493e-adc6-7e51e412154d\") " Nov 03 01:21:18.784203 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.784104 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-socket" pod "728c90f9-2248-493e-adc6-7e51e412154d" (UID: "728c90f9-2248-493e-adc6-7e51e412154d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:18.784320 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.784298 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-credential-socket" pod "728c90f9-2248-493e-adc6-7e51e412154d" (UID: "728c90f9-2248-493e-adc6-7e51e412154d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:18.784585 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.784520 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:18.784585 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.784521 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-data" pod "728c90f9-2248-493e-adc6-7e51e412154d" (UID: "728c90f9-2248-493e-adc6-7e51e412154d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:18.784585 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.784544 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:18.784937 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.784841 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-certs" pod "728c90f9-2248-493e-adc6-7e51e412154d" (UID: "728c90f9-2248-493e-adc6-7e51e412154d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:18.785168 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.785005 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/728c90f9-2248-493e-adc6-7e51e412154d-istiod-ca-cert" pod "728c90f9-2248-493e-adc6-7e51e412154d" (UID: "728c90f9-2248-493e-adc6-7e51e412154d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:18.786338 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.786285 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h"] Nov 03 01:21:18.787318 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.787293 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-kube-api-access-p62zw" pod "728c90f9-2248-493e-adc6-7e51e412154d" (UID: "728c90f9-2248-493e-adc6-7e51e412154d"). InnerVolumeSpecName "kube-api-access-p62zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:18.787629 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.787604 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/728c90f9-2248-493e-adc6-7e51e412154d-istio-podinfo" pod "728c90f9-2248-493e-adc6-7e51e412154d" (UID: "728c90f9-2248-493e-adc6-7e51e412154d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:21:18.787738 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.787714 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-envoy" pod "728c90f9-2248-493e-adc6-7e51e412154d" (UID: "728c90f9-2248-493e-adc6-7e51e412154d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:18.787806 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.787734 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-istio-token" pod "728c90f9-2248-493e-adc6-7e51e412154d" (UID: "728c90f9-2248-493e-adc6-7e51e412154d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:18.812288 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.812260 6400 scope.go:122] "RemoveContainer" containerID="4873956bad3c1b5220f2183e23dcdb004bd4e97b9c52b756a478ad1f7e9f8132" Nov 03 01:21:18.812633 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:21:18.812612 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4873956bad3c1b5220f2183e23dcdb004bd4e97b9c52b756a478ad1f7e9f8132\": container with ID starting with 4873956bad3c1b5220f2183e23dcdb004bd4e97b9c52b756a478ad1f7e9f8132 not found: ID does not exist" containerID="4873956bad3c1b5220f2183e23dcdb004bd4e97b9c52b756a478ad1f7e9f8132" Nov 03 01:21:18.812697 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.812646 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4873956bad3c1b5220f2183e23dcdb004bd4e97b9c52b756a478ad1f7e9f8132"} err="failed to get container status \"4873956bad3c1b5220f2183e23dcdb004bd4e97b9c52b756a478ad1f7e9f8132\": rpc error: code = NotFound desc = could not find container \"4873956bad3c1b5220f2183e23dcdb004bd4e97b9c52b756a478ad1f7e9f8132\": container with ID starting with 4873956bad3c1b5220f2183e23dcdb004bd4e97b9c52b756a478ad1f7e9f8132 not found: ID does not exist" Nov 03 01:21:18.812697 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.812665 6400 scope.go:122] "RemoveContainer" containerID="67935ffc9efc96bfe0d96a47b96cc311f1223647cb8edc5bb3f2b40d03a9dca3" Nov 03 01:21:18.812983 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:21:18.812966 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67935ffc9efc96bfe0d96a47b96cc311f1223647cb8edc5bb3f2b40d03a9dca3\": container with ID starting with 67935ffc9efc96bfe0d96a47b96cc311f1223647cb8edc5bb3f2b40d03a9dca3 not found: ID does not exist" containerID="67935ffc9efc96bfe0d96a47b96cc311f1223647cb8edc5bb3f2b40d03a9dca3" Nov 03 01:21:18.813058 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.812988 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67935ffc9efc96bfe0d96a47b96cc311f1223647cb8edc5bb3f2b40d03a9dca3"} err="failed to get container status \"67935ffc9efc96bfe0d96a47b96cc311f1223647cb8edc5bb3f2b40d03a9dca3\": rpc error: code = NotFound desc = could not find container \"67935ffc9efc96bfe0d96a47b96cc311f1223647cb8edc5bb3f2b40d03a9dca3\": container with ID starting with 67935ffc9efc96bfe0d96a47b96cc311f1223647cb8edc5bb3f2b40d03a9dca3 not found: ID does not exist" Nov 03 01:21:18.837251 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.837209 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8"] Nov 03 01:21:18.885285 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.885251 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:18.885285 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.885282 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728c90f9-2248-493e-adc6-7e51e412154d-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:18.885285 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.885294 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:18.885533 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.885302 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:18.885533 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.885311 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p62zw\" (UniqueName: \"kubernetes.io/projected/728c90f9-2248-493e-adc6-7e51e412154d-kube-api-access-p62zw\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:18.885533 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.885322 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728c90f9-2248-493e-adc6-7e51e412154d-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:18.885533 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.885330 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728c90f9-2248-493e-adc6-7e51e412154d-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:18.966994 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:18.966953 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:19.192410 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.192378 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk"] Nov 03 01:21:19.198023 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.197999 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20f0a3-openshift-default-74fdf645df-wbgnk"] Nov 03 01:21:19.208112 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.208060 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" podUID="d7683180-ca9e-4cd5-9956-60ea4baa8147" containerName="main" containerID="cri-o://77360d082d18eb855c41a87efee67a9611aa9be103e79fb1e9241a661b36cd18" gracePeriod=45 Nov 03 01:21:19.470275 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.470252 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:21:19.491793 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.491761 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-dshm\") pod \"d7683180-ca9e-4cd5-9956-60ea4baa8147\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " Nov 03 01:21:19.491988 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.491815 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-model-cache\") pod \"d7683180-ca9e-4cd5-9956-60ea4baa8147\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " Nov 03 01:21:19.491988 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.491904 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7683180-ca9e-4cd5-9956-60ea4baa8147-kube-api-access-p67nw\" (UniqueName: \"kubernetes.io/projected/d7683180-ca9e-4cd5-9956-60ea4baa8147-kube-api-access-p67nw\") pod \"d7683180-ca9e-4cd5-9956-60ea4baa8147\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " Nov 03 01:21:19.491988 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.491943 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7683180-ca9e-4cd5-9956-60ea4baa8147-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7683180-ca9e-4cd5-9956-60ea4baa8147-tls-certs\") pod \"d7683180-ca9e-4cd5-9956-60ea4baa8147\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " Nov 03 01:21:19.491988 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.491965 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-tmp-dir\") pod \"d7683180-ca9e-4cd5-9956-60ea4baa8147\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " Nov 03 01:21:19.492220 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.491994 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-home\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-home\") pod \"d7683180-ca9e-4cd5-9956-60ea4baa8147\" (UID: \"d7683180-ca9e-4cd5-9956-60ea4baa8147\") " Nov 03 01:21:19.492220 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.492086 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-model-cache" pod "d7683180-ca9e-4cd5-9956-60ea4baa8147" (UID: "d7683180-ca9e-4cd5-9956-60ea4baa8147"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:19.492341 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.492252 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-tmp-dir" pod "d7683180-ca9e-4cd5-9956-60ea4baa8147" (UID: "d7683180-ca9e-4cd5-9956-60ea4baa8147"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:19.492341 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.492283 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-home" pod "d7683180-ca9e-4cd5-9956-60ea4baa8147" (UID: "d7683180-ca9e-4cd5-9956-60ea4baa8147"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:19.492445 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.492429 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:19.492482 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.492449 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:19.492482 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.492459 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:19.494328 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.494299 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-dshm" pod "d7683180-ca9e-4cd5-9956-60ea4baa8147" (UID: "d7683180-ca9e-4cd5-9956-60ea4baa8147"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:19.494328 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.494303 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7683180-ca9e-4cd5-9956-60ea4baa8147-kube-api-access-p67nw" pod "d7683180-ca9e-4cd5-9956-60ea4baa8147" (UID: "d7683180-ca9e-4cd5-9956-60ea4baa8147"). InnerVolumeSpecName "kube-api-access-p67nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:19.494493 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.494308 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7683180-ca9e-4cd5-9956-60ea4baa8147-tls-certs" pod "d7683180-ca9e-4cd5-9956-60ea4baa8147" (UID: "d7683180-ca9e-4cd5-9956-60ea4baa8147"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:19.593157 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.593124 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7683180-ca9e-4cd5-9956-60ea4baa8147-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:19.593157 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.593153 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7683180-ca9e-4cd5-9956-60ea4baa8147-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:19.593157 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.593165 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p67nw\" (UniqueName: \"kubernetes.io/projected/d7683180-ca9e-4cd5-9956-60ea4baa8147-kube-api-access-p67nw\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:19.778906 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.778785 6400 generic.go:356] "Generic (PLEG): container finished" podID="d7683180-ca9e-4cd5-9956-60ea4baa8147" containerID="77360d082d18eb855c41a87efee67a9611aa9be103e79fb1e9241a661b36cd18" exitCode=0 Nov 03 01:21:19.778906 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.778858 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" Nov 03 01:21:19.778906 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.778868 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" event={"ID":"d7683180-ca9e-4cd5-9956-60ea4baa8147","Type":"ContainerDied","Data":"77360d082d18eb855c41a87efee67a9611aa9be103e79fb1e9241a661b36cd18"} Nov 03 01:21:19.779474 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.778922 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59" event={"ID":"d7683180-ca9e-4cd5-9956-60ea4baa8147","Type":"ContainerDied","Data":"000c90ff2a4f7a1d1d8e0558f4f01289a6298fe0dcf49f144c5052c45dd6c1d9"} Nov 03 01:21:19.779474 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.778945 6400 scope.go:122] "RemoveContainer" containerID="77360d082d18eb855c41a87efee67a9611aa9be103e79fb1e9241a661b36cd18" Nov 03 01:21:19.788623 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.788605 6400 scope.go:122] "RemoveContainer" containerID="77360d082d18eb855c41a87efee67a9611aa9be103e79fb1e9241a661b36cd18" Nov 03 01:21:19.788963 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:21:19.788943 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77360d082d18eb855c41a87efee67a9611aa9be103e79fb1e9241a661b36cd18\": container with ID starting with 77360d082d18eb855c41a87efee67a9611aa9be103e79fb1e9241a661b36cd18 not found: ID does not exist" containerID="77360d082d18eb855c41a87efee67a9611aa9be103e79fb1e9241a661b36cd18" Nov 03 01:21:19.789048 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.788977 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77360d082d18eb855c41a87efee67a9611aa9be103e79fb1e9241a661b36cd18"} err="failed to get container status \"77360d082d18eb855c41a87efee67a9611aa9be103e79fb1e9241a661b36cd18\": rpc error: code = NotFound desc = could not find container \"77360d082d18eb855c41a87efee67a9611aa9be103e79fb1e9241a661b36cd18\": container with ID starting with 77360d082d18eb855c41a87efee67a9611aa9be103e79fb1e9241a661b36cd18 not found: ID does not exist" Nov 03 01:21:19.940108 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.940076 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59"] Nov 03 01:21:19.943174 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:19.943147 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-904a34/sub-test-model-904a34-kserve-556796ccdc-ccm59"] Nov 03 01:21:21.178800 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:21.178766 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="728c90f9-2248-493e-adc6-7e51e412154d" path="/var/lib/kubelet/pods/728c90f9-2248-493e-adc6-7e51e412154d/volumes" Nov 03 01:21:21.179233 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:21.179218 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7683180-ca9e-4cd5-9956-60ea4baa8147" path="/var/lib/kubelet/pods/d7683180-ca9e-4cd5-9956-60ea4baa8147/volumes" Nov 03 01:21:21.852110 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:21.852068 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:23.582926 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:23.582894 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:24.345744 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:24.345706 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:25.150976 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:25.150938 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:27.152918 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:21:27.152880 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:21:28.272046 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:28.272004 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:21:28.331324 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:28.331268 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:31.297976 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:31.297939 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:31.471023 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:31.470988 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:31.648312 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:31.647685 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:31.893932 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:31.893892 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:32.096618 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:32.096581 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:33.030068 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:33.030036 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:33.353179 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:33.353139 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:33.567027 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:33.566992 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:33.809132 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:33.809094 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" podUID="50ce1488-84fe-4013-970a-83af8bea40b1" containerName="main" containerID="cri-o://3c25df44b88b64473a1fa43ccd0cc3eb53beaa3b3b3ed178fad8a805d7082ae2" gracePeriod=45 Nov 03 01:21:33.857855 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:33.857779 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" podUID="ca16bbd6-e0a8-4d24-ba33-a749b8659aee" containerName="main" containerID="cri-o://cb88806e94d9d9aecc98cdbea30808ef778ff428c8f89f0e22282cbafe66c4ad" gracePeriod=45 Nov 03 01:21:34.119017 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.118991 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 01:21:34.140015 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.139985 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 01:21:34.256048 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256009 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tmp-dir\") pod \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " Nov 03 01:21:34.256269 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256096 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-dshm\") pod \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " Nov 03 01:21:34.256269 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256185 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ce1488-84fe-4013-970a-83af8bea40b1-kube-api-access-b847w\" (UniqueName: \"kubernetes.io/projected/50ce1488-84fe-4013-970a-83af8bea40b1-kube-api-access-b847w\") pod \"50ce1488-84fe-4013-970a-83af8bea40b1\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " Nov 03 01:21:34.256269 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256224 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-model-cache\") pod \"50ce1488-84fe-4013-970a-83af8bea40b1\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " Nov 03 01:21:34.257501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256256 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tmp-dir" pod "ca16bbd6-e0a8-4d24-ba33-a749b8659aee" (UID: "ca16bbd6-e0a8-4d24-ba33-a749b8659aee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.257501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256266 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-model-cache\") pod \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " Nov 03 01:21:34.257501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256351 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-kube-api-access-m24ql\" (UniqueName: \"kubernetes.io/projected/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-kube-api-access-m24ql\") pod \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " Nov 03 01:21:34.257501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256392 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50ce1488-84fe-4013-970a-83af8bea40b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/50ce1488-84fe-4013-970a-83af8bea40b1-tls-certs\") pod \"50ce1488-84fe-4013-970a-83af8bea40b1\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " Nov 03 01:21:34.257501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256449 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-dshm\") pod \"50ce1488-84fe-4013-970a-83af8bea40b1\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " Nov 03 01:21:34.257501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256490 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-model-cache" pod "ca16bbd6-e0a8-4d24-ba33-a749b8659aee" (UID: "ca16bbd6-e0a8-4d24-ba33-a749b8659aee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.257501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256509 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-home\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-home\") pod \"50ce1488-84fe-4013-970a-83af8bea40b1\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " Nov 03 01:21:34.257501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256542 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-home\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-home\") pod \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " Nov 03 01:21:34.257501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256567 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tls-certs\") pod \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\" (UID: \"ca16bbd6-e0a8-4d24-ba33-a749b8659aee\") " Nov 03 01:21:34.257501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.256617 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-tmp-dir\") pod \"50ce1488-84fe-4013-970a-83af8bea40b1\" (UID: \"50ce1488-84fe-4013-970a-83af8bea40b1\") " Nov 03 01:21:34.257501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.257227 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-home" pod "50ce1488-84fe-4013-970a-83af8bea40b1" (UID: "50ce1488-84fe-4013-970a-83af8bea40b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.257501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.257314 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-home" pod "ca16bbd6-e0a8-4d24-ba33-a749b8659aee" (UID: "ca16bbd6-e0a8-4d24-ba33-a749b8659aee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.257501 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.257373 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-model-cache" pod "50ce1488-84fe-4013-970a-83af8bea40b1" (UID: "50ce1488-84fe-4013-970a-83af8bea40b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.258312 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.258261 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.258312 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.258282 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.258312 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.258283 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-tmp-dir" pod "50ce1488-84fe-4013-970a-83af8bea40b1" (UID: "50ce1488-84fe-4013-970a-83af8bea40b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.258312 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.258297 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.258312 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.258312 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.258601 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.258325 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.259292 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.259263 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50ce1488-84fe-4013-970a-83af8bea40b1-tls-certs" pod "50ce1488-84fe-4013-970a-83af8bea40b1" (UID: "50ce1488-84fe-4013-970a-83af8bea40b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:34.259468 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.259356 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-dshm" pod "ca16bbd6-e0a8-4d24-ba33-a749b8659aee" (UID: "ca16bbd6-e0a8-4d24-ba33-a749b8659aee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.259468 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.259397 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ce1488-84fe-4013-970a-83af8bea40b1-kube-api-access-b847w" pod "50ce1488-84fe-4013-970a-83af8bea40b1" (UID: "50ce1488-84fe-4013-970a-83af8bea40b1"). InnerVolumeSpecName "kube-api-access-b847w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:34.260674 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.260647 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-kube-api-access-m24ql" pod "ca16bbd6-e0a8-4d24-ba33-a749b8659aee" (UID: "ca16bbd6-e0a8-4d24-ba33-a749b8659aee"). InnerVolumeSpecName "kube-api-access-m24ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:34.261197 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.261162 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tls-certs" pod "ca16bbd6-e0a8-4d24-ba33-a749b8659aee" (UID: "ca16bbd6-e0a8-4d24-ba33-a749b8659aee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:34.261439 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.261415 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-dshm" pod "50ce1488-84fe-4013-970a-83af8bea40b1" (UID: "50ce1488-84fe-4013-970a-83af8bea40b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.300882 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.300837 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:34.359580 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.359480 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.359580 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.359508 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.359580 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.359518 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.359580 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.359531 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b847w\" (UniqueName: \"kubernetes.io/projected/50ce1488-84fe-4013-970a-83af8bea40b1-kube-api-access-b847w\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.359580 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.359547 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m24ql\" (UniqueName: \"kubernetes.io/projected/ca16bbd6-e0a8-4d24-ba33-a749b8659aee-kube-api-access-m24ql\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.359580 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.359561 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ce1488-84fe-4013-970a-83af8bea40b1-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.359580 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.359571 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ce1488-84fe-4013-970a-83af8bea40b1-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.865545 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.865505 6400 generic.go:356] "Generic (PLEG): container finished" podID="ca16bbd6-e0a8-4d24-ba33-a749b8659aee" containerID="cb88806e94d9d9aecc98cdbea30808ef778ff428c8f89f0e22282cbafe66c4ad" exitCode=0 Nov 03 01:21:34.865545 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.865548 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" Nov 03 01:21:34.865803 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.865543 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" event={"ID":"ca16bbd6-e0a8-4d24-ba33-a749b8659aee","Type":"ContainerDied","Data":"cb88806e94d9d9aecc98cdbea30808ef778ff428c8f89f0e22282cbafe66c4ad"} Nov 03 01:21:34.865803 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.865594 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8" event={"ID":"ca16bbd6-e0a8-4d24-ba33-a749b8659aee","Type":"ContainerDied","Data":"d253d311204662bb0309e52c00b88d9cdb6da4e928d4da1dce25f69bee9334fd"} Nov 03 01:21:34.865803 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.865619 6400 scope.go:122] "RemoveContainer" containerID="cb88806e94d9d9aecc98cdbea30808ef778ff428c8f89f0e22282cbafe66c4ad" Nov 03 01:21:34.867119 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.867097 6400 generic.go:356] "Generic (PLEG): container finished" podID="50ce1488-84fe-4013-970a-83af8bea40b1" containerID="3c25df44b88b64473a1fa43ccd0cc3eb53beaa3b3b3ed178fad8a805d7082ae2" exitCode=0 Nov 03 01:21:34.867224 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.867145 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" Nov 03 01:21:34.867224 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.867188 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" event={"ID":"50ce1488-84fe-4013-970a-83af8bea40b1","Type":"ContainerDied","Data":"3c25df44b88b64473a1fa43ccd0cc3eb53beaa3b3b3ed178fad8a805d7082ae2"} Nov 03 01:21:34.867300 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.867232 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" event={"ID":"50ce1488-84fe-4013-970a-83af8bea40b1","Type":"ContainerDied","Data":"d79c1ce23bfa69d5b1188aa6abf7d808a43c1a8baf9df2be3173691985c6f024"} Nov 03 01:21:34.880379 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.880195 6400 scope.go:122] "RemoveContainer" containerID="cb88806e94d9d9aecc98cdbea30808ef778ff428c8f89f0e22282cbafe66c4ad" Nov 03 01:21:34.881357 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:21:34.881315 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb88806e94d9d9aecc98cdbea30808ef778ff428c8f89f0e22282cbafe66c4ad\": container with ID starting with cb88806e94d9d9aecc98cdbea30808ef778ff428c8f89f0e22282cbafe66c4ad not found: ID does not exist" containerID="cb88806e94d9d9aecc98cdbea30808ef778ff428c8f89f0e22282cbafe66c4ad" Nov 03 01:21:34.881506 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.881366 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb88806e94d9d9aecc98cdbea30808ef778ff428c8f89f0e22282cbafe66c4ad"} err="failed to get container status \"cb88806e94d9d9aecc98cdbea30808ef778ff428c8f89f0e22282cbafe66c4ad\": rpc error: code = NotFound desc = could not find container \"cb88806e94d9d9aecc98cdbea30808ef778ff428c8f89f0e22282cbafe66c4ad\": container with ID starting with cb88806e94d9d9aecc98cdbea30808ef778ff428c8f89f0e22282cbafe66c4ad not found: ID does not exist" Nov 03 01:21:34.881582 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.881396 6400 scope.go:122] "RemoveContainer" containerID="3c25df44b88b64473a1fa43ccd0cc3eb53beaa3b3b3ed178fad8a805d7082ae2" Nov 03 01:21:34.898370 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.898342 6400 scope.go:122] "RemoveContainer" containerID="3c25df44b88b64473a1fa43ccd0cc3eb53beaa3b3b3ed178fad8a805d7082ae2" Nov 03 01:21:34.899046 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:21:34.899022 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c25df44b88b64473a1fa43ccd0cc3eb53beaa3b3b3ed178fad8a805d7082ae2\": container with ID starting with 3c25df44b88b64473a1fa43ccd0cc3eb53beaa3b3b3ed178fad8a805d7082ae2 not found: ID does not exist" containerID="3c25df44b88b64473a1fa43ccd0cc3eb53beaa3b3b3ed178fad8a805d7082ae2" Nov 03 01:21:34.899136 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:34.899056 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c25df44b88b64473a1fa43ccd0cc3eb53beaa3b3b3ed178fad8a805d7082ae2"} err="failed to get container status \"3c25df44b88b64473a1fa43ccd0cc3eb53beaa3b3b3ed178fad8a805d7082ae2\": rpc error: code = NotFound desc = could not find container \"3c25df44b88b64473a1fa43ccd0cc3eb53beaa3b3b3ed178fad8a805d7082ae2\": container with ID starting with 3c25df44b88b64473a1fa43ccd0cc3eb53beaa3b3b3ed178fad8a805d7082ae2 not found: ID does not exist" Nov 03 01:21:35.038461 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:35.038426 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8"] Nov 03 01:21:35.066695 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:35.066649 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h" podUID="50ce1488-84fe-4013-970a-83af8bea40b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:35.076664 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:35.076579 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-048135/simulator-w4-048135-kserve-54d59df6c8-6ntz8"] Nov 03 01:21:35.147675 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:35.147633 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h"] Nov 03 01:21:35.150910 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:35.150886 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-048135/distinct-2-w4-048135-kserve-55948649d5-s954h"] Nov 03 01:21:35.178386 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:35.178354 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50ce1488-84fe-4013-970a-83af8bea40b1" path="/var/lib/kubelet/pods/50ce1488-84fe-4013-970a-83af8bea40b1/volumes" Nov 03 01:21:35.178870 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:35.178815 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca16bbd6-e0a8-4d24-ba33-a749b8659aee" path="/var/lib/kubelet/pods/ca16bbd6-e0a8-4d24-ba33-a749b8659aee/volumes" Nov 03 01:21:35.472345 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:35.472251 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:35.653493 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:35.653457 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:37.110502 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:37.110460 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:38.312523 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:38.312489 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:39.607473 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:39.607438 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:21:40.433133 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:40.433093 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:41.348122 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:41.348084 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:41.496355 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:41.496271 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:42.608192 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:42.608152 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:43.888499 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:43.888461 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:44.486623 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:44.486588 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:45.621335 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:45.621298 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:46.169318 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:46.169277 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:49.986146 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:49.986109 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:50.193800 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:50.193756 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:50.399252 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:50.399214 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:55.461578 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:55.461541 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:55.636623 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:55.636588 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:56.228365 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:56.228320 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:56.390643 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:56.390604 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:57.152721 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:21:57.152676 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:21:58.028941 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:58.028910 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:58.534558 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:21:58.534524 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:00.332749 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:00.332711 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:00.541278 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:00.541242 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:01.472459 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:01.472424 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:01.742369 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:01.742280 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:01.937138 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:01.937104 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:22:02.220266 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:02.220232 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:02.480365 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:02.480276 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:02.658934 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:02.658900 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:02.827242 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:02.827200 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:02.984226 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:02.984193 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:03.161651 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:03.161569 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:06.633112 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:06.633070 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:07.652315 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:07.652280 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj"] Nov 03 01:22:07.701724 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:07.701683 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs"] Nov 03 01:22:07.703025 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:07.702999 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf"] Nov 03 01:22:07.782369 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:07.782334 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:11.415536 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:11.415500 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:14.083762 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:14.083720 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:16.121183 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:16.121145 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:20.414492 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:20.414452 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:20.558428 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:20.558398 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:22:21.568049 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:21.568010 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:21.715863 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:21.715799 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:22.673523 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:22.673475 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" podUID="1c78610a-8df0-4f6d-b980-cb697136ab3f" containerName="main" containerID="cri-o://ffe6a5150571234eaa49c9d4a067cf7cfc925330a78837a0f57985f048e2dbff" gracePeriod=45 Nov 03 01:22:22.722130 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:22.722084 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" podUID="2fd1b7e8-1357-47d7-9ba6-236e95443aae" containerName="main" containerID="cri-o://cc44c5285df5662c6df7ba541c4d2b184c91a4c7e76fb1452fa79f7522e2d00a" gracePeriod=45 Nov 03 01:22:22.722649 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:22.722621 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" podUID="1f7d333a-0851-4b86-a3ab-29b6ea453fa7" containerName="main" containerID="cri-o://d8ecc944a8982053963b7e548c16d9afd082a0e15a9dd7ee5f9d7ceeb845358a" gracePeriod=45 Nov 03 01:22:22.798544 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:22.798499 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" podUID="2fd1b7e8-1357-47d7-9ba6-236e95443aae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": dial tcp 10.134.0.47:8000: connect: connection refused" Nov 03 01:22:23.007938 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.007912 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 01:22:23.033702 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.033675 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 01:22:23.041689 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.041663 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 01:22:23.061897 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.061859 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-dshm\") pod \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " Nov 03 01:22:23.062074 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.061907 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-kube-api-access-g9vnp\" (UniqueName: \"kubernetes.io/projected/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-kube-api-access-g9vnp\") pod \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " Nov 03 01:22:23.062074 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.061943 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c78610a-8df0-4f6d-b980-cb697136ab3f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c78610a-8df0-4f6d-b980-cb697136ab3f-tls-certs\") pod \"1c78610a-8df0-4f6d-b980-cb697136ab3f\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " Nov 03 01:22:23.062074 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.061988 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd1b7e8-1357-47d7-9ba6-236e95443aae-kube-api-access-cxp52\" (UniqueName: \"kubernetes.io/projected/2fd1b7e8-1357-47d7-9ba6-236e95443aae-kube-api-access-cxp52\") pod \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " Nov 03 01:22:23.062074 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062017 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tmp-dir\") pod \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " Nov 03 01:22:23.062074 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062044 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-model-cache\") pod \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " Nov 03 01:22:23.062374 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062091 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-home\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-home\") pod \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " Nov 03 01:22:23.062374 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062117 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-dshm\") pod \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " Nov 03 01:22:23.062374 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062141 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-home\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-home\") pod \"1c78610a-8df0-4f6d-b980-cb697136ab3f\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " Nov 03 01:22:23.062374 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062184 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-dshm\") pod \"1c78610a-8df0-4f6d-b980-cb697136ab3f\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " Nov 03 01:22:23.062374 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062212 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tls-certs\") pod \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " Nov 03 01:22:23.062374 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062250 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tls-certs\") pod \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\" (UID: \"2fd1b7e8-1357-47d7-9ba6-236e95443aae\") " Nov 03 01:22:23.062374 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062313 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-model-cache\") pod \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " Nov 03 01:22:23.062374 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062338 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-model-cache\") pod \"1c78610a-8df0-4f6d-b980-cb697136ab3f\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " Nov 03 01:22:23.063000 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062390 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tmp-dir\") pod \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " Nov 03 01:22:23.063000 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062428 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-home\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-home\") pod \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\" (UID: \"1f7d333a-0851-4b86-a3ab-29b6ea453fa7\") " Nov 03 01:22:23.063000 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062457 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c78610a-8df0-4f6d-b980-cb697136ab3f-kube-api-access-g79pm\" (UniqueName: \"kubernetes.io/projected/1c78610a-8df0-4f6d-b980-cb697136ab3f-kube-api-access-g79pm\") pod \"1c78610a-8df0-4f6d-b980-cb697136ab3f\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " Nov 03 01:22:23.063000 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062504 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-tmp-dir\") pod \"1c78610a-8df0-4f6d-b980-cb697136ab3f\" (UID: \"1c78610a-8df0-4f6d-b980-cb697136ab3f\") " Nov 03 01:22:23.063000 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062676 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-model-cache" pod "1f7d333a-0851-4b86-a3ab-29b6ea453fa7" (UID: "1f7d333a-0851-4b86-a3ab-29b6ea453fa7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:23.063000 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062907 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.063440 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062986 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tmp-dir" pod "2fd1b7e8-1357-47d7-9ba6-236e95443aae" (UID: "2fd1b7e8-1357-47d7-9ba6-236e95443aae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:23.063521 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.062991 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-home" pod "1c78610a-8df0-4f6d-b980-cb697136ab3f" (UID: "1c78610a-8df0-4f6d-b980-cb697136ab3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:23.063521 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.063077 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-tmp-dir" pod "1c78610a-8df0-4f6d-b980-cb697136ab3f" (UID: "1c78610a-8df0-4f6d-b980-cb697136ab3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:23.063521 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.063276 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-model-cache" pod "1c78610a-8df0-4f6d-b980-cb697136ab3f" (UID: "1c78610a-8df0-4f6d-b980-cb697136ab3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:23.063521 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.063287 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-home" pod "1f7d333a-0851-4b86-a3ab-29b6ea453fa7" (UID: "1f7d333a-0851-4b86-a3ab-29b6ea453fa7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:23.063521 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.063326 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-model-cache" pod "2fd1b7e8-1357-47d7-9ba6-236e95443aae" (UID: "2fd1b7e8-1357-47d7-9ba6-236e95443aae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:23.063521 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.063369 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-home" pod "2fd1b7e8-1357-47d7-9ba6-236e95443aae" (UID: "2fd1b7e8-1357-47d7-9ba6-236e95443aae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:23.064052 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.063537 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tmp-dir" pod "1f7d333a-0851-4b86-a3ab-29b6ea453fa7" (UID: "1f7d333a-0851-4b86-a3ab-29b6ea453fa7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:23.065139 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.065113 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-dshm" pod "1f7d333a-0851-4b86-a3ab-29b6ea453fa7" (UID: "1f7d333a-0851-4b86-a3ab-29b6ea453fa7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:23.065240 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.065196 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd1b7e8-1357-47d7-9ba6-236e95443aae-kube-api-access-cxp52" pod "2fd1b7e8-1357-47d7-9ba6-236e95443aae" (UID: "2fd1b7e8-1357-47d7-9ba6-236e95443aae"). InnerVolumeSpecName "kube-api-access-cxp52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:23.065322 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.065267 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c78610a-8df0-4f6d-b980-cb697136ab3f-tls-certs" pod "1c78610a-8df0-4f6d-b980-cb697136ab3f" (UID: "1c78610a-8df0-4f6d-b980-cb697136ab3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:22:23.065548 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.065361 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-kube-api-access-g9vnp" pod "1f7d333a-0851-4b86-a3ab-29b6ea453fa7" (UID: "1f7d333a-0851-4b86-a3ab-29b6ea453fa7"). InnerVolumeSpecName "kube-api-access-g9vnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:23.066638 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.066616 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c78610a-8df0-4f6d-b980-cb697136ab3f-kube-api-access-g79pm" pod "1c78610a-8df0-4f6d-b980-cb697136ab3f" (UID: "1c78610a-8df0-4f6d-b980-cb697136ab3f"). InnerVolumeSpecName "kube-api-access-g79pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:23.066727 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.066664 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-dshm" pod "2fd1b7e8-1357-47d7-9ba6-236e95443aae" (UID: "2fd1b7e8-1357-47d7-9ba6-236e95443aae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:23.067328 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.067307 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tls-certs" pod "2fd1b7e8-1357-47d7-9ba6-236e95443aae" (UID: "2fd1b7e8-1357-47d7-9ba6-236e95443aae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:22:23.067543 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.067524 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-dshm" pod "1c78610a-8df0-4f6d-b980-cb697136ab3f" (UID: "1c78610a-8df0-4f6d-b980-cb697136ab3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:23.068113 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.068092 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tls-certs" pod "1f7d333a-0851-4b86-a3ab-29b6ea453fa7" (UID: "1f7d333a-0851-4b86-a3ab-29b6ea453fa7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:22:23.099381 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.099350 6400 generic.go:356] "Generic (PLEG): container finished" podID="1f7d333a-0851-4b86-a3ab-29b6ea453fa7" containerID="d8ecc944a8982053963b7e548c16d9afd082a0e15a9dd7ee5f9d7ceeb845358a" exitCode=0 Nov 03 01:22:23.099563 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.099408 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" Nov 03 01:22:23.099563 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.099421 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" event={"ID":"1f7d333a-0851-4b86-a3ab-29b6ea453fa7","Type":"ContainerDied","Data":"d8ecc944a8982053963b7e548c16d9afd082a0e15a9dd7ee5f9d7ceeb845358a"} Nov 03 01:22:23.099563 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.099460 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf" event={"ID":"1f7d333a-0851-4b86-a3ab-29b6ea453fa7","Type":"ContainerDied","Data":"82262f72200caa1d799bd73f01ec473bbd8c606e0bed5a8470e64fe774520548"} Nov 03 01:22:23.099563 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.099475 6400 scope.go:122] "RemoveContainer" containerID="d8ecc944a8982053963b7e548c16d9afd082a0e15a9dd7ee5f9d7ceeb845358a" Nov 03 01:22:23.100866 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.100840 6400 generic.go:356] "Generic (PLEG): container finished" podID="2fd1b7e8-1357-47d7-9ba6-236e95443aae" containerID="cc44c5285df5662c6df7ba541c4d2b184c91a4c7e76fb1452fa79f7522e2d00a" exitCode=0 Nov 03 01:22:23.100976 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.100875 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" event={"ID":"2fd1b7e8-1357-47d7-9ba6-236e95443aae","Type":"ContainerDied","Data":"cc44c5285df5662c6df7ba541c4d2b184c91a4c7e76fb1452fa79f7522e2d00a"} Nov 03 01:22:23.100976 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.100896 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" Nov 03 01:22:23.100976 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.100922 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs" event={"ID":"2fd1b7e8-1357-47d7-9ba6-236e95443aae","Type":"ContainerDied","Data":"00dfdc856a261d0f9bc44a775805bd0cc9f6b9da70bef02f4ecbcfc7c5b160f3"} Nov 03 01:22:23.102452 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.102431 6400 generic.go:356] "Generic (PLEG): container finished" podID="1c78610a-8df0-4f6d-b980-cb697136ab3f" containerID="ffe6a5150571234eaa49c9d4a067cf7cfc925330a78837a0f57985f048e2dbff" exitCode=0 Nov 03 01:22:23.102573 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.102470 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" event={"ID":"1c78610a-8df0-4f6d-b980-cb697136ab3f","Type":"ContainerDied","Data":"ffe6a5150571234eaa49c9d4a067cf7cfc925330a78837a0f57985f048e2dbff"} Nov 03 01:22:23.102573 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.102496 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" event={"ID":"1c78610a-8df0-4f6d-b980-cb697136ab3f","Type":"ContainerDied","Data":"ed334df6c3b061cc88f43be9f61a46b530f5ad6d142a8516e5909bccc368c57c"} Nov 03 01:22:23.102573 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.102545 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj" Nov 03 01:22:23.113423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.113405 6400 scope.go:122] "RemoveContainer" containerID="d8ecc944a8982053963b7e548c16d9afd082a0e15a9dd7ee5f9d7ceeb845358a" Nov 03 01:22:23.113793 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:22:23.113773 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ecc944a8982053963b7e548c16d9afd082a0e15a9dd7ee5f9d7ceeb845358a\": container with ID starting with d8ecc944a8982053963b7e548c16d9afd082a0e15a9dd7ee5f9d7ceeb845358a not found: ID does not exist" containerID="d8ecc944a8982053963b7e548c16d9afd082a0e15a9dd7ee5f9d7ceeb845358a" Nov 03 01:22:23.113922 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.113803 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ecc944a8982053963b7e548c16d9afd082a0e15a9dd7ee5f9d7ceeb845358a"} err="failed to get container status \"d8ecc944a8982053963b7e548c16d9afd082a0e15a9dd7ee5f9d7ceeb845358a\": rpc error: code = NotFound desc = could not find container \"d8ecc944a8982053963b7e548c16d9afd082a0e15a9dd7ee5f9d7ceeb845358a\": container with ID starting with d8ecc944a8982053963b7e548c16d9afd082a0e15a9dd7ee5f9d7ceeb845358a not found: ID does not exist" Nov 03 01:22:23.113922 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.113839 6400 scope.go:122] "RemoveContainer" containerID="cc44c5285df5662c6df7ba541c4d2b184c91a4c7e76fb1452fa79f7522e2d00a" Nov 03 01:22:23.125711 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.123024 6400 scope.go:122] "RemoveContainer" containerID="cc44c5285df5662c6df7ba541c4d2b184c91a4c7e76fb1452fa79f7522e2d00a" Nov 03 01:22:23.125711 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:22:23.123355 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc44c5285df5662c6df7ba541c4d2b184c91a4c7e76fb1452fa79f7522e2d00a\": container with ID starting with cc44c5285df5662c6df7ba541c4d2b184c91a4c7e76fb1452fa79f7522e2d00a not found: ID does not exist" containerID="cc44c5285df5662c6df7ba541c4d2b184c91a4c7e76fb1452fa79f7522e2d00a" Nov 03 01:22:23.125711 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.123382 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc44c5285df5662c6df7ba541c4d2b184c91a4c7e76fb1452fa79f7522e2d00a"} err="failed to get container status \"cc44c5285df5662c6df7ba541c4d2b184c91a4c7e76fb1452fa79f7522e2d00a\": rpc error: code = NotFound desc = could not find container \"cc44c5285df5662c6df7ba541c4d2b184c91a4c7e76fb1452fa79f7522e2d00a\": container with ID starting with cc44c5285df5662c6df7ba541c4d2b184c91a4c7e76fb1452fa79f7522e2d00a not found: ID does not exist" Nov 03 01:22:23.125711 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.123400 6400 scope.go:122] "RemoveContainer" containerID="ffe6a5150571234eaa49c9d4a067cf7cfc925330a78837a0f57985f048e2dbff" Nov 03 01:22:23.134334 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.134313 6400 scope.go:122] "RemoveContainer" containerID="ffe6a5150571234eaa49c9d4a067cf7cfc925330a78837a0f57985f048e2dbff" Nov 03 01:22:23.134690 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:22:23.134663 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe6a5150571234eaa49c9d4a067cf7cfc925330a78837a0f57985f048e2dbff\": container with ID starting with ffe6a5150571234eaa49c9d4a067cf7cfc925330a78837a0f57985f048e2dbff not found: ID does not exist" containerID="ffe6a5150571234eaa49c9d4a067cf7cfc925330a78837a0f57985f048e2dbff" Nov 03 01:22:23.134765 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.134702 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe6a5150571234eaa49c9d4a067cf7cfc925330a78837a0f57985f048e2dbff"} err="failed to get container status \"ffe6a5150571234eaa49c9d4a067cf7cfc925330a78837a0f57985f048e2dbff\": rpc error: code = NotFound desc = could not find container \"ffe6a5150571234eaa49c9d4a067cf7cfc925330a78837a0f57985f048e2dbff\": container with ID starting with ffe6a5150571234eaa49c9d4a067cf7cfc925330a78837a0f57985f048e2dbff not found: ID does not exist" Nov 03 01:22:23.164392 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164360 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164392 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164390 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9vnp\" (UniqueName: \"kubernetes.io/projected/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-kube-api-access-g9vnp\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164392 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164406 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c78610a-8df0-4f6d-b980-cb697136ab3f-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164392 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164421 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxp52\" (UniqueName: \"kubernetes.io/projected/2fd1b7e8-1357-47d7-9ba6-236e95443aae-kube-api-access-cxp52\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164460 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164472 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164484 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164496 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd1b7e8-1357-47d7-9ba6-236e95443aae-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164507 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164517 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-dshm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164527 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164538 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd1b7e8-1357-47d7-9ba6-236e95443aae-tls-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164550 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-model-cache\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164566 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164577 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f7d333a-0851-4b86-a3ab-29b6ea453fa7-home\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164588 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g79pm\" (UniqueName: \"kubernetes.io/projected/1c78610a-8df0-4f6d-b980-cb697136ab3f-kube-api-access-g79pm\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.164760 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.164600 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c78610a-8df0-4f6d-b980-cb697136ab3f-tmp-dir\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.292299 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.292268 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs"] Nov 03 01:22:23.296758 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.296718 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/premium-w1-e28e1a-kserve-759df99987-279zs"] Nov 03 01:22:23.402620 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.402587 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf"] Nov 03 01:22:23.408469 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.408443 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/simulator-w1-e28e1a-kserve-6674b5c878-q6tpf"] Nov 03 01:22:23.577256 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.577185 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj"] Nov 03 01:22:23.580919 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:23.580893 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e28e1a/distinct-2-w1-e28e1a-kserve-5f75bc9545-z7fsj"] Nov 03 01:22:24.707609 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:24.707575 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:25.177978 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:25.177943 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c78610a-8df0-4f6d-b980-cb697136ab3f" path="/var/lib/kubelet/pods/1c78610a-8df0-4f6d-b980-cb697136ab3f/volumes" Nov 03 01:22:25.178500 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:25.178472 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f7d333a-0851-4b86-a3ab-29b6ea453fa7" path="/var/lib/kubelet/pods/1f7d333a-0851-4b86-a3ab-29b6ea453fa7/volumes" Nov 03 01:22:25.178902 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:25.178887 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd1b7e8-1357-47d7-9ba6-236e95443aae" path="/var/lib/kubelet/pods/2fd1b7e8-1357-47d7-9ba6-236e95443aae/volumes" Nov 03 01:22:26.335027 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:26.334970 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:27.154085 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:22:27.154045 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:22:32.020182 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:32.020097 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:35.339461 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.339430 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf"] Nov 03 01:22:35.339884 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.339632 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" podUID="0806992d-fe7f-49d4-80d8-78ceb1bccf72" containerName="maas-api" containerID="cri-o://bbe3b8f9e6ee177aa839f2a9e62aad0fef6a87406744e043bbedee2ea6601933" gracePeriod=30 Nov 03 01:22:35.463989 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.463953 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:35.582262 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.582226 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:22:35.618385 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.618350 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 01:22:35.682258 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.682231 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-metrics-certs\") pod \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " Nov 03 01:22:35.682471 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.682267 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0806992d-fe7f-49d4-80d8-78ceb1bccf72-kube-api-access-jcp62\" (UniqueName: \"kubernetes.io/projected/0806992d-fe7f-49d4-80d8-78ceb1bccf72-kube-api-access-jcp62\") pod \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " Nov 03 01:22:35.682471 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.682294 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0806992d-fe7f-49d4-80d8-78ceb1bccf72-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0806992d-fe7f-49d4-80d8-78ceb1bccf72-openshift-service-ca\") pod \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " Nov 03 01:22:35.682471 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.682383 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-maas-api-tls\") pod \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\" (UID: \"0806992d-fe7f-49d4-80d8-78ceb1bccf72\") " Nov 03 01:22:35.682733 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.682700 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0806992d-fe7f-49d4-80d8-78ceb1bccf72-openshift-service-ca" pod "0806992d-fe7f-49d4-80d8-78ceb1bccf72" (UID: "0806992d-fe7f-49d4-80d8-78ceb1bccf72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:35.684623 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.684587 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0806992d-fe7f-49d4-80d8-78ceb1bccf72-kube-api-access-jcp62" pod "0806992d-fe7f-49d4-80d8-78ceb1bccf72" (UID: "0806992d-fe7f-49d4-80d8-78ceb1bccf72"). InnerVolumeSpecName "kube-api-access-jcp62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:35.684705 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.684676 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-maas-api-tls" pod "0806992d-fe7f-49d4-80d8-78ceb1bccf72" (UID: "0806992d-fe7f-49d4-80d8-78ceb1bccf72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:22:35.684834 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.684801 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-metrics-certs" pod "0806992d-fe7f-49d4-80d8-78ceb1bccf72" (UID: "0806992d-fe7f-49d4-80d8-78ceb1bccf72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:22:35.784191 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.784160 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:35.784191 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.784187 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0806992d-fe7f-49d4-80d8-78ceb1bccf72-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:35.784191 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.784198 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcp62\" (UniqueName: \"kubernetes.io/projected/0806992d-fe7f-49d4-80d8-78ceb1bccf72-kube-api-access-jcp62\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:35.784446 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:35.784208 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0806992d-fe7f-49d4-80d8-78ceb1bccf72-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:36.170939 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:36.170906 6400 generic.go:356] "Generic (PLEG): container finished" podID="0806992d-fe7f-49d4-80d8-78ceb1bccf72" containerID="bbe3b8f9e6ee177aa839f2a9e62aad0fef6a87406744e043bbedee2ea6601933" exitCode=0 Nov 03 01:22:36.171129 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:36.170954 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" Nov 03 01:22:36.171129 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:36.170986 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" event={"ID":"0806992d-fe7f-49d4-80d8-78ceb1bccf72","Type":"ContainerDied","Data":"bbe3b8f9e6ee177aa839f2a9e62aad0fef6a87406744e043bbedee2ea6601933"} Nov 03 01:22:36.171129 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:36.171049 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf" event={"ID":"0806992d-fe7f-49d4-80d8-78ceb1bccf72","Type":"ContainerDied","Data":"4478d8e040d17531a11136f13855be9305b9c5fad42725848268d8b9ed73ad5f"} Nov 03 01:22:36.171129 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:36.171073 6400 scope.go:122] "RemoveContainer" containerID="bbe3b8f9e6ee177aa839f2a9e62aad0fef6a87406744e043bbedee2ea6601933" Nov 03 01:22:36.180626 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:36.180609 6400 scope.go:122] "RemoveContainer" containerID="bbe3b8f9e6ee177aa839f2a9e62aad0fef6a87406744e043bbedee2ea6601933" Nov 03 01:22:36.180929 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:22:36.180912 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe3b8f9e6ee177aa839f2a9e62aad0fef6a87406744e043bbedee2ea6601933\": container with ID starting with bbe3b8f9e6ee177aa839f2a9e62aad0fef6a87406744e043bbedee2ea6601933 not found: ID does not exist" containerID="bbe3b8f9e6ee177aa839f2a9e62aad0fef6a87406744e043bbedee2ea6601933" Nov 03 01:22:36.180992 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:36.180938 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe3b8f9e6ee177aa839f2a9e62aad0fef6a87406744e043bbedee2ea6601933"} err="failed to get container status \"bbe3b8f9e6ee177aa839f2a9e62aad0fef6a87406744e043bbedee2ea6601933\": rpc error: code = NotFound desc = could not find container \"bbe3b8f9e6ee177aa839f2a9e62aad0fef6a87406744e043bbedee2ea6601933\": container with ID starting with bbe3b8f9e6ee177aa839f2a9e62aad0fef6a87406744e043bbedee2ea6601933 not found: ID does not exist" Nov 03 01:22:36.332415 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:36.332378 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf"] Nov 03 01:22:36.335929 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:36.335902 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e28e1a-78bdd98f96-7crkf"] Nov 03 01:22:37.177236 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:37.177208 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0806992d-fe7f-49d4-80d8-78ceb1bccf72" path="/var/lib/kubelet/pods/0806992d-fe7f-49d4-80d8-78ceb1bccf72/volumes" Nov 03 01:22:37.803063 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:37.803030 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:38.025531 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:38.025496 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:39.235011 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:39.234979 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr"] Nov 03 01:22:39.235457 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:39.235175 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerName="istio-proxy" containerID="cri-o://b082280d6ca603c7478241b3d153aff3c2bbe3321943b932bbd6343e949e2313" gracePeriod=30 Nov 03 01:22:39.284422 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:39.284390 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25"] Nov 03 01:22:39.284608 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:39.284589 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" containerName="istio-proxy" containerID="cri-o://982c7e7e1820757336f60c78bc0361a5d0deca679cf3d7705e05c19a5969126e" gracePeriod=30 Nov 03 01:22:40.714136 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:40.714097 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:22:40.714521 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:40.714163 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:22:40.885256 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:40.885219 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:22:40.885441 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:40.885275 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:22:43.877387 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:43.877347 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:44.036568 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.036527 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:44.533931 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.533894 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:22:44.559850 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.559809 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:22:44.665439 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665398 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-data\") pod \"6c17cab1-0794-408d-9d68-bd68708e3d39\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " Nov 03 01:22:44.665619 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665457 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-certs\") pod \"26ce4a34-1291-40bc-b3de-95ad93828326\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " Nov 03 01:22:44.665619 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665501 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-envoy\") pod \"26ce4a34-1291-40bc-b3de-95ad93828326\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " Nov 03 01:22:44.665619 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665541 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-kube-api-access-smdh6\" (UniqueName: \"kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-kube-api-access-smdh6\") pod \"26ce4a34-1291-40bc-b3de-95ad93828326\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " Nov 03 01:22:44.665619 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665575 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-socket\") pod \"6c17cab1-0794-408d-9d68-bd68708e3d39\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " Nov 03 01:22:44.665899 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665639 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-credential-socket\") pod \"26ce4a34-1291-40bc-b3de-95ad93828326\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " Nov 03 01:22:44.665899 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665676 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-data" pod "6c17cab1-0794-408d-9d68-bd68708e3d39" (UID: "6c17cab1-0794-408d-9d68-bd68708e3d39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:44.665899 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665690 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-podinfo\") pod \"6c17cab1-0794-408d-9d68-bd68708e3d39\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " Nov 03 01:22:44.665899 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665689 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-certs" pod "26ce4a34-1291-40bc-b3de-95ad93828326" (UID: "26ce4a34-1291-40bc-b3de-95ad93828326"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:44.665899 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665758 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ce4a34-1291-40bc-b3de-95ad93828326-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ce4a34-1291-40bc-b3de-95ad93828326-istiod-ca-cert\") pod \"26ce4a34-1291-40bc-b3de-95ad93828326\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " Nov 03 01:22:44.665899 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665810 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-socket" pod "6c17cab1-0794-408d-9d68-bd68708e3d39" (UID: "6c17cab1-0794-408d-9d68-bd68708e3d39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:44.665899 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665812 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c17cab1-0794-408d-9d68-bd68708e3d39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17cab1-0794-408d-9d68-bd68708e3d39-istiod-ca-cert\") pod \"6c17cab1-0794-408d-9d68-bd68708e3d39\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " Nov 03 01:22:44.665899 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665884 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-token\" (UniqueName: \"kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-token\") pod \"6c17cab1-0794-408d-9d68-bd68708e3d39\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " Nov 03 01:22:44.665899 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665885 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-credential-socket" pod "26ce4a34-1291-40bc-b3de-95ad93828326" (UID: "26ce4a34-1291-40bc-b3de-95ad93828326"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:44.666555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665921 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-credential-socket\") pod \"6c17cab1-0794-408d-9d68-bd68708e3d39\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " Nov 03 01:22:44.666555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.665978 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-certs\") pod \"6c17cab1-0794-408d-9d68-bd68708e3d39\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " Nov 03 01:22:44.666555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.666011 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-kube-api-access-8bn5g\" (UniqueName: \"kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-kube-api-access-8bn5g\") pod \"6c17cab1-0794-408d-9d68-bd68708e3d39\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " Nov 03 01:22:44.666555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.666047 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-envoy\") pod \"6c17cab1-0794-408d-9d68-bd68708e3d39\" (UID: \"6c17cab1-0794-408d-9d68-bd68708e3d39\") " Nov 03 01:22:44.666555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.666075 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-istio-token\" (UniqueName: \"kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-istio-token\") pod \"26ce4a34-1291-40bc-b3de-95ad93828326\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " Nov 03 01:22:44.666555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.666105 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-socket\") pod \"26ce4a34-1291-40bc-b3de-95ad93828326\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " Nov 03 01:22:44.666555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.666121 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c17cab1-0794-408d-9d68-bd68708e3d39-istiod-ca-cert" pod "6c17cab1-0794-408d-9d68-bd68708e3d39" (UID: "6c17cab1-0794-408d-9d68-bd68708e3d39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:44.666555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.666137 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-data\") pod \"26ce4a34-1291-40bc-b3de-95ad93828326\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " Nov 03 01:22:44.666555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.666162 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ce4a34-1291-40bc-b3de-95ad93828326-istiod-ca-cert" pod "26ce4a34-1291-40bc-b3de-95ad93828326" (UID: "26ce4a34-1291-40bc-b3de-95ad93828326"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:44.666555 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.666174 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26ce4a34-1291-40bc-b3de-95ad93828326-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ce4a34-1291-40bc-b3de-95ad93828326-istio-podinfo\") pod \"26ce4a34-1291-40bc-b3de-95ad93828326\" (UID: \"26ce4a34-1291-40bc-b3de-95ad93828326\") " Nov 03 01:22:44.667145 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.666881 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.667145 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.666903 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.667145 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.666919 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.667145 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.666933 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ce4a34-1291-40bc-b3de-95ad93828326-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.667145 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.667097 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17cab1-0794-408d-9d68-bd68708e3d39-istiod-ca-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.667145 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.667114 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.667464 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.667281 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-certs" pod "6c17cab1-0794-408d-9d68-bd68708e3d39" (UID: "6c17cab1-0794-408d-9d68-bd68708e3d39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:44.667464 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.667436 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-socket" pod "26ce4a34-1291-40bc-b3de-95ad93828326" (UID: "26ce4a34-1291-40bc-b3de-95ad93828326"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:44.667576 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.667550 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-credential-socket" pod "6c17cab1-0794-408d-9d68-bd68708e3d39" (UID: "6c17cab1-0794-408d-9d68-bd68708e3d39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:44.667638 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.667615 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-data" pod "26ce4a34-1291-40bc-b3de-95ad93828326" (UID: "26ce4a34-1291-40bc-b3de-95ad93828326"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:44.668919 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.668475 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-kube-api-access-smdh6" pod "26ce4a34-1291-40bc-b3de-95ad93828326" (UID: "26ce4a34-1291-40bc-b3de-95ad93828326"). InnerVolumeSpecName "kube-api-access-smdh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:44.668919 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.668578 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-envoy" pod "26ce4a34-1291-40bc-b3de-95ad93828326" (UID: "26ce4a34-1291-40bc-b3de-95ad93828326"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:44.669120 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.669002 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-kube-api-access-8bn5g" pod "6c17cab1-0794-408d-9d68-bd68708e3d39" (UID: "6c17cab1-0794-408d-9d68-bd68708e3d39"). InnerVolumeSpecName "kube-api-access-8bn5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:44.669549 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.669519 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-istio-token" pod "26ce4a34-1291-40bc-b3de-95ad93828326" (UID: "26ce4a34-1291-40bc-b3de-95ad93828326"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:44.669549 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.669531 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-token" pod "6c17cab1-0794-408d-9d68-bd68708e3d39" (UID: "6c17cab1-0794-408d-9d68-bd68708e3d39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:44.669740 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.669721 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26ce4a34-1291-40bc-b3de-95ad93828326-istio-podinfo" pod "26ce4a34-1291-40bc-b3de-95ad93828326" (UID: "26ce4a34-1291-40bc-b3de-95ad93828326"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:22:44.670019 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.670002 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-podinfo" pod "6c17cab1-0794-408d-9d68-bd68708e3d39" (UID: "6c17cab1-0794-408d-9d68-bd68708e3d39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:22:44.670457 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.670445 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-envoy" pod "6c17cab1-0794-408d-9d68-bd68708e3d39" (UID: "6c17cab1-0794-408d-9d68-bd68708e3d39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:44.767755 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.767681 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.767755 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.767707 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smdh6\" (UniqueName: \"kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-kube-api-access-smdh6\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.767755 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.767718 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.767755 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.767728 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.767755 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.767738 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-credential-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.767755 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.767747 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-workload-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.767755 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.767757 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bn5g\" (UniqueName: \"kubernetes.io/projected/6c17cab1-0794-408d-9d68-bd68708e3d39-kube-api-access-8bn5g\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.768120 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.767766 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17cab1-0794-408d-9d68-bd68708e3d39-istio-envoy\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.768120 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.767775 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ce4a34-1291-40bc-b3de-95ad93828326-istio-token\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.768120 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.767783 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-workload-socket\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.768120 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.767791 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ce4a34-1291-40bc-b3de-95ad93828326-istio-data\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:44.768120 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:44.767799 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ce4a34-1291-40bc-b3de-95ad93828326-istio-podinfo\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:45.212455 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.212421 6400 generic.go:356] "Generic (PLEG): container finished" podID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerID="b082280d6ca603c7478241b3d153aff3c2bbe3321943b932bbd6343e949e2313" exitCode=0 Nov 03 01:22:45.212919 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.212470 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" Nov 03 01:22:45.212919 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.212507 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" event={"ID":"6c17cab1-0794-408d-9d68-bd68708e3d39","Type":"ContainerDied","Data":"b082280d6ca603c7478241b3d153aff3c2bbe3321943b932bbd6343e949e2313"} Nov 03 01:22:45.212919 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.212563 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr" event={"ID":"6c17cab1-0794-408d-9d68-bd68708e3d39","Type":"ContainerDied","Data":"7f4a575c6f2b3cfa7be15acd5df667d8059825b1d465e184ba5519a23ba3db60"} Nov 03 01:22:45.212919 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.212591 6400 scope.go:122] "RemoveContainer" containerID="b082280d6ca603c7478241b3d153aff3c2bbe3321943b932bbd6343e949e2313" Nov 03 01:22:45.213938 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.213919 6400 generic.go:356] "Generic (PLEG): container finished" podID="26ce4a34-1291-40bc-b3de-95ad93828326" containerID="982c7e7e1820757336f60c78bc0361a5d0deca679cf3d7705e05c19a5969126e" exitCode=0 Nov 03 01:22:45.214055 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.213991 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" event={"ID":"26ce4a34-1291-40bc-b3de-95ad93828326","Type":"ContainerDied","Data":"982c7e7e1820757336f60c78bc0361a5d0deca679cf3d7705e05c19a5969126e"} Nov 03 01:22:45.214055 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.214035 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" event={"ID":"26ce4a34-1291-40bc-b3de-95ad93828326","Type":"ContainerDied","Data":"88df3c1a72dd484fc0b9cee727e1b085e387c4ec6fe15c7588e8e3d6ad359700"} Nov 03 01:22:45.214055 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.214037 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25" Nov 03 01:22:45.222015 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.221994 6400 scope.go:122] "RemoveContainer" containerID="b082280d6ca603c7478241b3d153aff3c2bbe3321943b932bbd6343e949e2313" Nov 03 01:22:45.222272 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:22:45.222251 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b082280d6ca603c7478241b3d153aff3c2bbe3321943b932bbd6343e949e2313\": container with ID starting with b082280d6ca603c7478241b3d153aff3c2bbe3321943b932bbd6343e949e2313 not found: ID does not exist" containerID="b082280d6ca603c7478241b3d153aff3c2bbe3321943b932bbd6343e949e2313" Nov 03 01:22:45.222365 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.222285 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b082280d6ca603c7478241b3d153aff3c2bbe3321943b932bbd6343e949e2313"} err="failed to get container status \"b082280d6ca603c7478241b3d153aff3c2bbe3321943b932bbd6343e949e2313\": rpc error: code = NotFound desc = could not find container \"b082280d6ca603c7478241b3d153aff3c2bbe3321943b932bbd6343e949e2313\": container with ID starting with b082280d6ca603c7478241b3d153aff3c2bbe3321943b932bbd6343e949e2313 not found: ID does not exist" Nov 03 01:22:45.222365 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.222305 6400 scope.go:122] "RemoveContainer" containerID="982c7e7e1820757336f60c78bc0361a5d0deca679cf3d7705e05c19a5969126e" Nov 03 01:22:45.230025 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.230009 6400 scope.go:122] "RemoveContainer" containerID="982c7e7e1820757336f60c78bc0361a5d0deca679cf3d7705e05c19a5969126e" Nov 03 01:22:45.230298 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:22:45.230276 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"982c7e7e1820757336f60c78bc0361a5d0deca679cf3d7705e05c19a5969126e\": container with ID starting with 982c7e7e1820757336f60c78bc0361a5d0deca679cf3d7705e05c19a5969126e not found: ID does not exist" containerID="982c7e7e1820757336f60c78bc0361a5d0deca679cf3d7705e05c19a5969126e" Nov 03 01:22:45.230371 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.230305 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"982c7e7e1820757336f60c78bc0361a5d0deca679cf3d7705e05c19a5969126e"} err="failed to get container status \"982c7e7e1820757336f60c78bc0361a5d0deca679cf3d7705e05c19a5969126e\": rpc error: code = NotFound desc = could not find container \"982c7e7e1820757336f60c78bc0361a5d0deca679cf3d7705e05c19a5969126e\": container with ID starting with 982c7e7e1820757336f60c78bc0361a5d0deca679cf3d7705e05c19a5969126e not found: ID does not exist" Nov 03 01:22:45.375003 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.374972 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25"] Nov 03 01:22:45.424886 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.424849 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bzd25"] Nov 03 01:22:45.488534 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.488451 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr"] Nov 03 01:22:45.494448 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.494421 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-r7lxr"] Nov 03 01:22:45.512832 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:45.512788 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:47.184637 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:47.179551 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" path="/var/lib/kubelet/pods/26ce4a34-1291-40bc-b3de-95ad93828326/volumes" Nov 03 01:22:47.184637 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:47.180221 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" path="/var/lib/kubelet/pods/6c17cab1-0794-408d-9d68-bd68708e3d39/volumes" Nov 03 01:22:48.077152 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:48.077116 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:48.616558 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:48.616517 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:50.616810 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:50.616771 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:22:51.485896 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:51.485862 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:51.680052 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:51.680022 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:52.790967 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:52.790939 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-8d9rl_fc80ce75-cd2c-43c0-888b-4e41bbd9e047/limitador/0.log" Nov 03 01:22:54.420578 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:54.420552 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-62pxm_288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1/manager/0.log" Nov 03 01:22:54.978606 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:54.978564 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bg4db_9adacad7-56f7-41dc-90c2-0dbdab888504/manager/0.log" Nov 03 01:22:54.988522 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:54.988488 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:55.141097 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:55.141059 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:55.582287 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:22:55.582241 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-785d4477d7-99w47_13b7d7ab-cb01-490e-98fb-86d5fb539fb7/manager/0.log" Nov 03 01:22:57.151171 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:22:57.151134 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:23:00.838130 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:00.838093 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:01.469924 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:01.469869 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:01.635665 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:01.635627 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:04.600225 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:04.600193 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:04.813574 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:04.813536 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:07.868416 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:07.868383 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:08.067110 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:08.067081 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:10.503997 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:10.503955 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:10.769754 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:10.769662 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:10.931175 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:10.931133 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:11.088580 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:11.088541 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:15.612199 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:15.612161 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:18.658581 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:18.658539 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:23:20.420161 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:20.417640 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:25.053688 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:25.053655 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:25.294117 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:25.294079 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:27.151073 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:23:27.151037 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:23:30.111722 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:30.111689 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:35.129842 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:35.129787 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:37.705755 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:37.705723 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:39.908171 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:39.908137 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:43.001103 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:43.001062 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:43.211979 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:43.211939 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:44.638038 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:44.638000 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:48.837507 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:48.837460 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:50.300923 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:50.300888 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:50.574116 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:50.574030 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:52.299918 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:52.299880 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:54.951990 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:54.951944 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:23:56.568462 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:56.568429 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-785d4477d7-99w47"] Nov 03 01:23:56.568951 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:56.568628 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-785d4477d7-99w47" podUID="13b7d7ab-cb01-490e-98fb-86d5fb539fb7" containerName="manager" containerID="cri-o://2fc8dbde268d07914731e6e9e7193ab1ed3c252fe56b8f9a872e2de0083c5b1d" gracePeriod=30 Nov 03 01:23:56.819738 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:56.819680 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 01:23:56.923788 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:56.923746 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-webhook-cert\" (UniqueName: \"kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-webhook-cert\") pod \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\" (UID: \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\") " Nov 03 01:23:56.924011 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:56.923852 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-metrics-certs\") pod \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\" (UID: \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\") " Nov 03 01:23:56.924011 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:56.923953 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-kube-api-access-hl22s\" (UniqueName: \"kubernetes.io/projected/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-kube-api-access-hl22s\") pod \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\" (UID: \"13b7d7ab-cb01-490e-98fb-86d5fb539fb7\") " Nov 03 01:23:56.926138 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:56.926113 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-webhook-cert" pod "13b7d7ab-cb01-490e-98fb-86d5fb539fb7" (UID: "13b7d7ab-cb01-490e-98fb-86d5fb539fb7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:23:56.926260 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:56.926137 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-kube-api-access-hl22s" pod "13b7d7ab-cb01-490e-98fb-86d5fb539fb7" (UID: "13b7d7ab-cb01-490e-98fb-86d5fb539fb7"). InnerVolumeSpecName "kube-api-access-hl22s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:23:56.926260 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:56.926157 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-metrics-certs" pod "13b7d7ab-cb01-490e-98fb-86d5fb539fb7" (UID: "13b7d7ab-cb01-490e-98fb-86d5fb539fb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:23:57.025230 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:57.025197 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:57.025230 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:57.025227 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl22s\" (UniqueName: \"kubernetes.io/projected/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-kube-api-access-hl22s\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:57.025230 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:57.025237 6400 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13b7d7ab-cb01-490e-98fb-86d5fb539fb7-webhook-cert\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:57.151074 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:23:57.150994 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:23:57.544736 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:57.544704 6400 generic.go:356] "Generic (PLEG): container finished" podID="13b7d7ab-cb01-490e-98fb-86d5fb539fb7" containerID="2fc8dbde268d07914731e6e9e7193ab1ed3c252fe56b8f9a872e2de0083c5b1d" exitCode=0 Nov 03 01:23:57.544950 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:57.544758 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785d4477d7-99w47" Nov 03 01:23:57.544950 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:57.544801 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785d4477d7-99w47" event={"ID":"13b7d7ab-cb01-490e-98fb-86d5fb539fb7","Type":"ContainerDied","Data":"2fc8dbde268d07914731e6e9e7193ab1ed3c252fe56b8f9a872e2de0083c5b1d"} Nov 03 01:23:57.544950 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:57.544868 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785d4477d7-99w47" event={"ID":"13b7d7ab-cb01-490e-98fb-86d5fb539fb7","Type":"ContainerDied","Data":"73052efc5187e9ad0cb315aa0648238a8ef656cd7445aae8e8deba14fabeed2d"} Nov 03 01:23:57.544950 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:57.544896 6400 scope.go:122] "RemoveContainer" containerID="2fc8dbde268d07914731e6e9e7193ab1ed3c252fe56b8f9a872e2de0083c5b1d" Nov 03 01:23:57.554766 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:57.554748 6400 scope.go:122] "RemoveContainer" containerID="2fc8dbde268d07914731e6e9e7193ab1ed3c252fe56b8f9a872e2de0083c5b1d" Nov 03 01:23:57.555032 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:23:57.555012 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc8dbde268d07914731e6e9e7193ab1ed3c252fe56b8f9a872e2de0083c5b1d\": container with ID starting with 2fc8dbde268d07914731e6e9e7193ab1ed3c252fe56b8f9a872e2de0083c5b1d not found: ID does not exist" containerID="2fc8dbde268d07914731e6e9e7193ab1ed3c252fe56b8f9a872e2de0083c5b1d" Nov 03 01:23:57.555084 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:57.555049 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc8dbde268d07914731e6e9e7193ab1ed3c252fe56b8f9a872e2de0083c5b1d"} err="failed to get container status \"2fc8dbde268d07914731e6e9e7193ab1ed3c252fe56b8f9a872e2de0083c5b1d\": rpc error: code = NotFound desc = could not find container \"2fc8dbde268d07914731e6e9e7193ab1ed3c252fe56b8f9a872e2de0083c5b1d\": container with ID starting with 2fc8dbde268d07914731e6e9e7193ab1ed3c252fe56b8f9a872e2de0083c5b1d not found: ID does not exist" Nov 03 01:23:57.707245 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:57.707215 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-785d4477d7-99w47"] Nov 03 01:23:57.756806 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:57.756754 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-785d4477d7-99w47"] Nov 03 01:23:59.177777 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:23:59.177747 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b7d7ab-cb01-490e-98fb-86d5fb539fb7" path="/var/lib/kubelet/pods/13b7d7ab-cb01-490e-98fb-86d5fb539fb7/volumes" Nov 03 01:24:25.387968 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:24:25.387930 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:27.150542 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:24:27.150507 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:24:30.571891 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:24:30.571858 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:31.948242 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:24:31.948204 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:35.486021 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:24:35.485993 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:37.505792 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:24:37.505754 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:49.466544 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:24:49.466503 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:49.634674 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:24:49.634639 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:51.473838 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:24:51.473779 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:57.151210 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:24:57.151167 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:25:02.747528 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:02.747491 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:25:21.045067 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:21.045039 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:25:25.486976 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.486943 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs"] Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487275 6400 cpu_manager.go:414] "containerMap: removing container" podUID="eaf4988a-ac9f-446e-91ec-28ee63f944a7" containerName="main" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487286 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf4988a-ac9f-446e-91ec-28ee63f944a7" containerName="main" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487298 6400 cpu_manager.go:414] "containerMap: removing container" podUID="5faaa2d3-41ef-42c6-a47c-8af1daeb5798" containerName="maas-api" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487302 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5faaa2d3-41ef-42c6-a47c-8af1daeb5798" containerName="maas-api" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487309 6400 cpu_manager.go:414] "containerMap: removing container" podUID="728c90f9-2248-493e-adc6-7e51e412154d" containerName="istio-proxy" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487314 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728c90f9-2248-493e-adc6-7e51e412154d" containerName="istio-proxy" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487323 6400 cpu_manager.go:414] "containerMap: removing container" podUID="89d21cbe-9619-4d67-bf52-105653fa5e73" containerName="main" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487329 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d21cbe-9619-4d67-bf52-105653fa5e73" containerName="main" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487337 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487341 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487346 6400 cpu_manager.go:414] "containerMap: removing container" podUID="48320114-ae7f-4b99-9aec-cc27e247fc4d" containerName="maas-api" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487350 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48320114-ae7f-4b99-9aec-cc27e247fc4d" containerName="maas-api" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487357 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" containerName="cleanup" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487361 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" containerName="cleanup" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487366 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerName="istio-proxy" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487370 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerName="istio-proxy" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487378 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" containerName="cleanup" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487382 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" containerName="cleanup" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487388 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f60672ea-e415-4096-bf4f-4689d4194dbf" containerName="main" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487392 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f60672ea-e415-4096-bf4f-4689d4194dbf" containerName="main" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487398 6400 cpu_manager.go:414] "containerMap: removing container" podUID="09687b53-7262-438c-a249-123aefe4ccf8" containerName="cleanup" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487402 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09687b53-7262-438c-a249-123aefe4ccf8" containerName="cleanup" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487408 6400 cpu_manager.go:414] "containerMap: removing container" podUID="29761a2e-1646-4996-9f3e-89a0ace1588d" containerName="maas-api" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487412 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29761a2e-1646-4996-9f3e-89a0ace1588d" containerName="maas-api" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487421 6400 cpu_manager.go:414] "containerMap: removing container" podUID="bc8bfcd4-3359-4a5f-aeef-50516cf73fa3" containerName="maas-api" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487425 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8bfcd4-3359-4a5f-aeef-50516cf73fa3" containerName="maas-api" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487431 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c5e04815-50f0-4f14-b1fd-73a120e09378" containerName="cleanup" Nov 03 01:25:25.487423 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487435 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e04815-50f0-4f14-b1fd-73a120e09378" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487442 6400 cpu_manager.go:414] "containerMap: removing container" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" containerName="istio-proxy" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487446 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" containerName="istio-proxy" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487452 6400 cpu_manager.go:414] "containerMap: removing container" podUID="d7683180-ca9e-4cd5-9956-60ea4baa8147" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487457 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7683180-ca9e-4cd5-9956-60ea4baa8147" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487463 6400 cpu_manager.go:414] "containerMap: removing container" podUID="2fd1b7e8-1357-47d7-9ba6-236e95443aae" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487467 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd1b7e8-1357-47d7-9ba6-236e95443aae" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487472 6400 cpu_manager.go:414] "containerMap: removing container" podUID="50ce1488-84fe-4013-970a-83af8bea40b1" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487476 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ce1488-84fe-4013-970a-83af8bea40b1" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487484 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487488 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487494 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487498 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487505 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c5e04815-50f0-4f14-b1fd-73a120e09378" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487509 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e04815-50f0-4f14-b1fd-73a120e09378" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487520 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0806992d-fe7f-49d4-80d8-78ceb1bccf72" containerName="maas-api" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487524 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0806992d-fe7f-49d4-80d8-78ceb1bccf72" containerName="maas-api" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487529 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1f7d333a-0851-4b86-a3ab-29b6ea453fa7" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487533 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7d333a-0851-4b86-a3ab-29b6ea453fa7" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487539 6400 cpu_manager.go:414] "containerMap: removing container" podUID="09687b53-7262-438c-a249-123aefe4ccf8" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487543 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09687b53-7262-438c-a249-123aefe4ccf8" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487549 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487553 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487558 6400 cpu_manager.go:414] "containerMap: removing container" podUID="13b7d7ab-cb01-490e-98fb-86d5fb539fb7" containerName="manager" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487562 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b7d7ab-cb01-490e-98fb-86d5fb539fb7" containerName="manager" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487570 6400 cpu_manager.go:414] "containerMap: removing container" podUID="ca16bbd6-e0a8-4d24-ba33-a749b8659aee" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487573 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca16bbd6-e0a8-4d24-ba33-a749b8659aee" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487578 6400 cpu_manager.go:414] "containerMap: removing container" podUID="09687b53-7262-438c-a249-123aefe4ccf8" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487582 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09687b53-7262-438c-a249-123aefe4ccf8" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487590 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c5e04815-50f0-4f14-b1fd-73a120e09378" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487593 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e04815-50f0-4f14-b1fd-73a120e09378" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487603 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a9fc68ce-c4aa-4e6f-b265-8802ee46de54" containerName="maas-api" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487607 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fc68ce-c4aa-4e6f-b265-8802ee46de54" containerName="maas-api" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487616 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1c78610a-8df0-4f6d-b980-cb697136ab3f" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487620 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c78610a-8df0-4f6d-b980-cb697136ab3f" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487626 6400 cpu_manager.go:414] "containerMap: removing container" podUID="dfbe3d21-ff1e-4d7e-8e16-705478b6f95d" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487630 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbe3d21-ff1e-4d7e-8e16-705478b6f95d" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487637 6400 cpu_manager.go:414] "containerMap: removing container" podUID="03e891a7-9472-4300-b9f0-b75a745280b9" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487642 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e891a7-9472-4300-b9f0-b75a745280b9" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487648 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487652 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487700 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fc68ce-c4aa-4e6f-b265-8802ee46de54" containerName="maas-api" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487709 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c78610a-8df0-4f6d-b980-cb697136ab3f" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487715 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbe3d21-ff1e-4d7e-8e16-705478b6f95d" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487721 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e891a7-9472-4300-b9f0-b75a745280b9" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487727 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487733 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf4988a-ac9f-446e-91ec-28ee63f944a7" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487758 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="728c90f9-2248-493e-adc6-7e51e412154d" containerName="istio-proxy" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487763 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="5faaa2d3-41ef-42c6-a47c-8af1daeb5798" containerName="maas-api" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487769 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d21cbe-9619-4d67-bf52-105653fa5e73" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487773 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487778 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="48320114-ae7f-4b99-9aec-cc27e247fc4d" containerName="maas-api" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487783 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487788 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c17cab1-0794-408d-9d68-bd68708e3d39" containerName="istio-proxy" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487792 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487799 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f60672ea-e415-4096-bf4f-4689d4194dbf" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487804 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="09687b53-7262-438c-a249-123aefe4ccf8" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487809 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="29761a2e-1646-4996-9f3e-89a0ace1588d" containerName="maas-api" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487814 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8bfcd4-3359-4a5f-aeef-50516cf73fa3" containerName="maas-api" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487832 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e04815-50f0-4f14-b1fd-73a120e09378" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487837 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ce4a34-1291-40bc-b3de-95ad93828326" containerName="istio-proxy" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487842 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7683180-ca9e-4cd5-9956-60ea4baa8147" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487850 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd1b7e8-1357-47d7-9ba6-236e95443aae" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487858 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ce1488-84fe-4013-970a-83af8bea40b1" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487865 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487870 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e04815-50f0-4f14-b1fd-73a120e09378" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487875 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0806992d-fe7f-49d4-80d8-78ceb1bccf72" containerName="maas-api" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487879 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7d333a-0851-4b86-a3ab-29b6ea453fa7" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487884 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="09687b53-7262-438c-a249-123aefe4ccf8" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487888 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487894 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b7d7ab-cb01-490e-98fb-86d5fb539fb7" containerName="manager" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487899 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca16bbd6-e0a8-4d24-ba33-a749b8659aee" containerName="main" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487904 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="09687b53-7262-438c-a249-123aefe4ccf8" containerName="cleanup" Nov 03 01:25:25.488326 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.487909 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e04815-50f0-4f14-b1fd-73a120e09378" containerName="cleanup" Nov 03 01:25:25.490947 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.490933 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.536506 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.536481 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm"] Nov 03 01:25:25.536804 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.536792 6400 cpu_manager.go:414] "containerMap: removing container" podUID="728c90f9-2248-493e-adc6-7e51e412154d" containerName="istio-proxy" Nov 03 01:25:25.536895 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.536805 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728c90f9-2248-493e-adc6-7e51e412154d" containerName="istio-proxy" Nov 03 01:25:25.536895 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.536833 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" containerName="cleanup" Nov 03 01:25:25.536895 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.536838 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0dee74-1c25-495c-bf82-d6d155f42c2f" containerName="cleanup" Nov 03 01:25:25.536995 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.536917 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e42c2bc-a7d9-480b-b717-6411c0da5b8c" containerName="cleanup" Nov 03 01:25:25.536995 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.536925 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="728c90f9-2248-493e-adc6-7e51e412154d" containerName="istio-proxy" Nov 03 01:25:25.536995 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.536932 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df412b7-1ae9-4f96-a92a-85e29370dabb" containerName="istio-proxy" Nov 03 01:25:25.539869 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.539855 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.592755 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.592730 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs"] Nov 03 01:25:25.597761 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.597742 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.597847 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.597779 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.597909 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.597890 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.597972 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.597961 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.598009 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.597992 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.598044 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.598020 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.598115 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.598103 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.598153 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.598125 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.598190 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.598170 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nn6h\" (UniqueName: \"kubernetes.io/projected/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-kube-api-access-6nn6h\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.648918 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.648889 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm"] Nov 03 01:25:25.699528 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.699503 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.699667 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.699545 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.699667 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.699574 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.699667 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.699603 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.699813 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.699709 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.699813 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.699792 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k7bp\" (UniqueName: \"kubernetes.io/projected/b2266845-4cd0-4d7e-b244-1f6bfd536847-kube-api-access-5k7bp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.699965 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.699892 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.699965 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.699940 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.699965 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.699945 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nn6h\" (UniqueName: \"kubernetes.io/projected/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-kube-api-access-6nn6h\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.700093 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.699991 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.700093 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.700011 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.700093 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.700043 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2266845-4cd0-4d7e-b244-1f6bfd536847-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.700093 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.700060 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.700093 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.700085 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2266845-4cd0-4d7e-b244-1f6bfd536847-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.700266 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.700134 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.700266 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.700159 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.700266 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.700208 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2266845-4cd0-4d7e-b244-1f6bfd536847-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.700266 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.700245 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.700406 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.700275 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.700406 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.700302 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.700406 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.700319 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.700406 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.700386 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.700795 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.700780 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.702208 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.702185 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.702811 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.702794 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.748043 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.747984 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:25.755631 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.755604 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.755957 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.755939 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nn6h\" (UniqueName: \"kubernetes.io/projected/ceaafa9a-74d0-40cf-8324-7c4f3cdd663f-kube-api-access-6nn6h\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs\" (UID: \"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.800949 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.800920 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.800949 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.800949 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.801126 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.800964 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2266845-4cd0-4d7e-b244-1f6bfd536847-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.801126 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.800992 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.801126 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.801035 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.801126 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.801051 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7bp\" (UniqueName: \"kubernetes.io/projected/b2266845-4cd0-4d7e-b244-1f6bfd536847-kube-api-access-5k7bp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.801126 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.801093 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.801368 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.801149 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2266845-4cd0-4d7e-b244-1f6bfd536847-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.801368 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.801179 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2266845-4cd0-4d7e-b244-1f6bfd536847-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.801368 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.801291 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.801590 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.801571 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.801693 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.801670 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.801750 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.801678 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.802027 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.802006 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2266845-4cd0-4d7e-b244-1f6bfd536847-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.803103 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.803085 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:25.803290 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.803272 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2266845-4cd0-4d7e-b244-1f6bfd536847-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.803861 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.803844 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2266845-4cd0-4d7e-b244-1f6bfd536847-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.863858 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.861078 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2266845-4cd0-4d7e-b244-1f6bfd536847-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:25.863858 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:25.861729 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7bp\" (UniqueName: \"kubernetes.io/projected/b2266845-4cd0-4d7e-b244-1f6bfd536847-kube-api-access-5k7bp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsttm\" (UID: \"b2266845-4cd0-4d7e-b244-1f6bfd536847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:26.151412 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:26.151372 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:26.223940 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:26.223856 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs"] Nov 03 01:25:26.231593 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:25:26.231566 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceaafa9a_74d0_40cf_8324_7c4f3cdd663f.slice/crio-e3c3cac17ef19519b7f28f20736927e40e5591ec5de73d911ecf2d2e428d01e5 WatchSource:0}: Error finding container e3c3cac17ef19519b7f28f20736927e40e5591ec5de73d911ecf2d2e428d01e5: Status 404 returned error can't find the container with id e3c3cac17ef19519b7f28f20736927e40e5591ec5de73d911ecf2d2e428d01e5 Nov 03 01:25:26.234314 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:26.233796 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:25:26.234314 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:26.233909 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:25:26.234314 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:26.233954 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:25:26.528757 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:26.528727 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm"] Nov 03 01:25:26.530879 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:25:26.530853 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2266845_4cd0_4d7e_b244_1f6bfd536847.slice/crio-c56667d1db854b9a49fd5ac4fc61ae90864ca59e8a805ecf46a7bd58448c8dc9 WatchSource:0}: Error finding container c56667d1db854b9a49fd5ac4fc61ae90864ca59e8a805ecf46a7bd58448c8dc9: Status 404 returned error can't find the container with id c56667d1db854b9a49fd5ac4fc61ae90864ca59e8a805ecf46a7bd58448c8dc9 Nov 03 01:25:26.533409 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:26.533382 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:25:26.533482 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:26.533442 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:25:26.533482 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:26.533478 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 01:25:26.932486 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:26.931994 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" event={"ID":"b2266845-4cd0-4d7e-b244-1f6bfd536847","Type":"ContainerStarted","Data":"cf00475773abc6da4781ca392fdfbcaf5dd5e963b2b74de13c36748eee72d9d3"} Nov 03 01:25:26.932486 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:26.932064 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" event={"ID":"b2266845-4cd0-4d7e-b244-1f6bfd536847","Type":"ContainerStarted","Data":"c56667d1db854b9a49fd5ac4fc61ae90864ca59e8a805ecf46a7bd58448c8dc9"} Nov 03 01:25:26.934496 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:26.934465 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" event={"ID":"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f","Type":"ContainerStarted","Data":"30cfab0db7fe27440ec6f3bbb973e42b179f62a7e1ee161fff5a0cf111d557b8"} Nov 03 01:25:26.934706 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:26.934689 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" event={"ID":"ceaafa9a-74d0-40cf-8324-7c4f3cdd663f","Type":"ContainerStarted","Data":"e3c3cac17ef19519b7f28f20736927e40e5591ec5de73d911ecf2d2e428d01e5"} Nov 03 01:25:27.051516 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:27.051110 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" podStartSLOduration=2.051090511 podStartE2EDuration="2.051090511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:27.049578396 +0000 UTC m=+4774.000518554" watchObservedRunningTime="2026-11-03 01:25:27.051090511 +0000 UTC m=+4774.002030672" Nov 03 01:25:27.154932 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:27.154898 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:27.155893 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:27.155753 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 01:25:27.155893 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:27.155813 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" podUID="b2266845-4cd0-4d7e-b244-1f6bfd536847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 01:25:27.169354 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:27.169308 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" podStartSLOduration=2.169295321 podStartE2EDuration="2.169295321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:27.168960582 +0000 UTC m=+4774.119900738" watchObservedRunningTime="2026-11-03 01:25:27.169295321 +0000 UTC m=+4774.120235480" Nov 03 01:25:27.222110 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:25:27.221997 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:25:27.812229 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:27.812196 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:27.821016 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:27.820967 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 01:25:27.821319 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:27.821271 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" podUID="ceaafa9a-74d0-40cf-8324-7c4f3cdd663f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 01:25:28.152603 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:28.152516 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 01:25:28.152603 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:28.152589 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" podUID="b2266845-4cd0-4d7e-b244-1f6bfd536847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 01:25:28.810322 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:28.810267 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 01:25:28.810621 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:28.810592 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" podUID="ceaafa9a-74d0-40cf-8324-7c4f3cdd663f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 01:25:29.152135 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:29.152055 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 01:25:29.152582 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:29.152551 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" podUID="b2266845-4cd0-4d7e-b244-1f6bfd536847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 01:25:29.807845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:29.804177 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 01:25:29.807845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:29.804233 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" podUID="ceaafa9a-74d0-40cf-8324-7c4f3cdd663f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 01:25:30.152392 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:30.152307 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 01:25:30.152392 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:30.152361 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" podUID="b2266845-4cd0-4d7e-b244-1f6bfd536847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 01:25:30.772491 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:30.772458 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:25:30.805270 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:30.805230 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 01:25:30.805564 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:30.805537 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" podUID="ceaafa9a-74d0-40cf-8324-7c4f3cdd663f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 01:25:31.152369 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:31.152336 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 01:25:31.152697 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:31.152676 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" podUID="b2266845-4cd0-4d7e-b244-1f6bfd536847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 01:25:31.807918 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:31.807878 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 01:25:31.808212 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:31.808182 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" podUID="ceaafa9a-74d0-40cf-8324-7c4f3cdd663f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 01:25:32.152591 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:32.152500 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 01:25:32.152591 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:32.152555 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" podUID="b2266845-4cd0-4d7e-b244-1f6bfd536847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 01:25:32.804145 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:32.804113 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 01:25:32.804365 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:32.804161 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" podUID="ceaafa9a-74d0-40cf-8324-7c4f3cdd663f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 01:25:33.152159 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:33.152063 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 01:25:33.152159 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:33.152114 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" podUID="b2266845-4cd0-4d7e-b244-1f6bfd536847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 01:25:34.308086 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:34.308053 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:34.309231 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:34.309207 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:34.310007 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:34.309982 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" Nov 03 01:25:34.742113 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:34.742035 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:34.742461 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:34.742439 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:34.743410 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:34.743392 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsttm" Nov 03 01:25:40.937179 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:40.937143 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:25:49.720276 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:25:49.720244 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:25:57.153679 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:25:57.153641 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:26:06.678642 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:26:06.678610 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:06.843063 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:26:06.843029 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:20.519081 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:26:20.519044 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:26.139089 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:26:26.139053 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:27.150485 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:26:27.150452 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:26:28.323484 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:26:28.323446 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:26:31.475540 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:26:31.475506 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:32.181211 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:26:32.181177 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:35.858411 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:26:35.858368 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:42.232687 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:26:42.232653 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:26:57.151575 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:26:57.151529 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:27:14.207477 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:14.207374 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:27:19.271514 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:19.271477 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:25.302537 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.302500 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f"] Nov 03 01:27:25.305857 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.305841 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" Nov 03 01:27:25.410155 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.410125 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:27:25.410336 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.410315 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:27:25.410405 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.410332 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:27:25.410467 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.410457 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:27:25.411488 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.411466 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f"] Nov 03 01:27:25.420438 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.420419 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478f117a-96k6f\" (UID: \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" Nov 03 01:27:25.420536 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.420491 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssh8c\" (UniqueName: \"kubernetes.io/projected/7082571b-9ab9-4a36-a0c5-e2a5a9510917-kube-api-access-ssh8c\") pod \"maas-api-revoke-sub-478f117a-96k6f\" (UID: \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" Nov 03 01:27:25.420597 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.420544 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478f117a-96k6f\" (UID: \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" Nov 03 01:27:25.521130 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.521099 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssh8c\" (UniqueName: \"kubernetes.io/projected/7082571b-9ab9-4a36-a0c5-e2a5a9510917-kube-api-access-ssh8c\") pod \"maas-api-revoke-sub-478f117a-96k6f\" (UID: \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" Nov 03 01:27:25.521283 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.521156 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478f117a-96k6f\" (UID: \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" Nov 03 01:27:25.521283 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.521224 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478f117a-96k6f\" (UID: \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" Nov 03 01:27:25.521810 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.521786 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478f117a-96k6f\" (UID: \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" Nov 03 01:27:25.523684 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.523667 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478f117a-96k6f\" (UID: \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" Nov 03 01:27:25.577843 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.577765 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssh8c\" (UniqueName: \"kubernetes.io/projected/7082571b-9ab9-4a36-a0c5-e2a5a9510917-kube-api-access-ssh8c\") pod \"maas-api-revoke-sub-478f117a-96k6f\" (UID: \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" Nov 03 01:27:25.617687 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.617661 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" Nov 03 01:27:25.987629 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:25.987551 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f"] Nov 03 01:27:25.990109 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:27:25.990085 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7082571b_9ab9_4a36_a0c5_e2a5a9510917.slice/crio-322b2cb885386dd4b2285101b467cb067b89b2fb95620539b5171d5369b2fb31 WatchSource:0}: Error finding container 322b2cb885386dd4b2285101b467cb067b89b2fb95620539b5171d5369b2fb31: Status 404 returned error can't find the container with id 322b2cb885386dd4b2285101b467cb067b89b2fb95620539b5171d5369b2fb31 Nov 03 01:27:26.435396 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:26.435361 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:26.561179 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:26.561104 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" event={"ID":"7082571b-9ab9-4a36-a0c5-e2a5a9510917","Type":"ContainerStarted","Data":"3c67174fc7979e357d295bf4cc8dc599f701eda99f00d9fbcfedcfff1bd4df6a"} Nov 03 01:27:26.561179 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:26.561150 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" event={"ID":"7082571b-9ab9-4a36-a0c5-e2a5a9510917","Type":"ContainerStarted","Data":"322b2cb885386dd4b2285101b467cb067b89b2fb95620539b5171d5369b2fb31"} Nov 03 01:27:26.674369 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:26.674317 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" podStartSLOduration=1.674302452 podStartE2EDuration="1.674302452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:26.673744972 +0000 UTC m=+4893.624685131" watchObservedRunningTime="2026-11-03 01:27:26.674302452 +0000 UTC m=+4893.625242634" Nov 03 01:27:27.151993 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:27:27.151961 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:27:27.567018 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:27.566982 6400 generic.go:356] "Generic (PLEG): container finished" podID="7082571b-9ab9-4a36-a0c5-e2a5a9510917" containerID="3c67174fc7979e357d295bf4cc8dc599f701eda99f00d9fbcfedcfff1bd4df6a" exitCode=0 Nov 03 01:27:27.567411 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:27.567064 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" event={"ID":"7082571b-9ab9-4a36-a0c5-e2a5a9510917","Type":"ContainerDied","Data":"3c67174fc7979e357d295bf4cc8dc599f701eda99f00d9fbcfedcfff1bd4df6a"} Nov 03 01:27:28.700717 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:28.700685 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" Nov 03 01:27:28.752006 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:28.751966 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-xks-ca\") pod \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\" (UID: \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\") " Nov 03 01:27:28.752160 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:28.752092 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-openshift-ca\") pod \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\" (UID: \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\") " Nov 03 01:27:28.752221 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:28.752161 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7082571b-9ab9-4a36-a0c5-e2a5a9510917-kube-api-access-ssh8c\" (UniqueName: \"kubernetes.io/projected/7082571b-9ab9-4a36-a0c5-e2a5a9510917-kube-api-access-ssh8c\") pod \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\" (UID: \"7082571b-9ab9-4a36-a0c5-e2a5a9510917\") " Nov 03 01:27:28.752494 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:28.752468 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-openshift-ca" pod "7082571b-9ab9-4a36-a0c5-e2a5a9510917" (UID: "7082571b-9ab9-4a36-a0c5-e2a5a9510917"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:27:28.754157 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:28.754131 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-xks-ca" pod "7082571b-9ab9-4a36-a0c5-e2a5a9510917" (UID: "7082571b-9ab9-4a36-a0c5-e2a5a9510917"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:27:28.754315 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:28.754299 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7082571b-9ab9-4a36-a0c5-e2a5a9510917-kube-api-access-ssh8c" pod "7082571b-9ab9-4a36-a0c5-e2a5a9510917" (UID: "7082571b-9ab9-4a36-a0c5-e2a5a9510917"). InnerVolumeSpecName "kube-api-access-ssh8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:27:28.853196 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:28.853113 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssh8c\" (UniqueName: \"kubernetes.io/projected/7082571b-9ab9-4a36-a0c5-e2a5a9510917-kube-api-access-ssh8c\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:27:28.853196 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:28.853150 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-xks-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:27:28.853196 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:28.853161 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7082571b-9ab9-4a36-a0c5-e2a5a9510917-maas-api-openshift-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:27:29.576686 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:29.576658 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" Nov 03 01:27:29.576686 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:29.576669 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f" event={"ID":"7082571b-9ab9-4a36-a0c5-e2a5a9510917","Type":"ContainerDied","Data":"322b2cb885386dd4b2285101b467cb067b89b2fb95620539b5171d5369b2fb31"} Nov 03 01:27:29.576940 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:29.576755 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="322b2cb885386dd4b2285101b467cb067b89b2fb95620539b5171d5369b2fb31" Nov 03 01:27:38.249315 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:38.249277 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:27:38.511360 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:38.511253 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:38.661362 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:38.661325 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:41.808735 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:41.808696 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:43.969766 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:43.969733 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf"] Nov 03 01:27:43.970208 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:43.970127 6400 cpu_manager.go:414] "containerMap: removing container" podUID="7082571b-9ab9-4a36-a0c5-e2a5a9510917" containerName="revoke-keys" Nov 03 01:27:43.970208 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:43.970137 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7082571b-9ab9-4a36-a0c5-e2a5a9510917" containerName="revoke-keys" Nov 03 01:27:43.970303 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:43.970215 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7082571b-9ab9-4a36-a0c5-e2a5a9510917" containerName="revoke-keys" Nov 03 01:27:43.973215 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:43.973200 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" Nov 03 01:27:44.076572 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.076534 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:27:44.076572 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.076560 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:27:44.076572 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.076544 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:27:44.076873 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.076607 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:27:44.078197 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.078176 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf"] Nov 03 01:27:44.098573 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.098547 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd26c0ac-2pthf\" (UID: \"904d622c-9c62-4b50-8d74-df32398e7f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" Nov 03 01:27:44.098716 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.098594 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd26c0ac-2pthf\" (UID: \"904d622c-9c62-4b50-8d74-df32398e7f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" Nov 03 01:27:44.098716 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.098701 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq4md\" (UniqueName: \"kubernetes.io/projected/904d622c-9c62-4b50-8d74-df32398e7f4a-kube-api-access-rq4md\") pod \"maas-api-revoke-sub-cd26c0ac-2pthf\" (UID: \"904d622c-9c62-4b50-8d74-df32398e7f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" Nov 03 01:27:44.199989 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.199933 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4md\" (UniqueName: \"kubernetes.io/projected/904d622c-9c62-4b50-8d74-df32398e7f4a-kube-api-access-rq4md\") pod \"maas-api-revoke-sub-cd26c0ac-2pthf\" (UID: \"904d622c-9c62-4b50-8d74-df32398e7f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" Nov 03 01:27:44.200163 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.200023 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd26c0ac-2pthf\" (UID: \"904d622c-9c62-4b50-8d74-df32398e7f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" Nov 03 01:27:44.200163 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.200085 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd26c0ac-2pthf\" (UID: \"904d622c-9c62-4b50-8d74-df32398e7f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" Nov 03 01:27:44.200880 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.200751 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd26c0ac-2pthf\" (UID: \"904d622c-9c62-4b50-8d74-df32398e7f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" Nov 03 01:27:44.202503 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.202481 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd26c0ac-2pthf\" (UID: \"904d622c-9c62-4b50-8d74-df32398e7f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" Nov 03 01:27:44.256411 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.256343 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4md\" (UniqueName: \"kubernetes.io/projected/904d622c-9c62-4b50-8d74-df32398e7f4a-kube-api-access-rq4md\") pod \"maas-api-revoke-sub-cd26c0ac-2pthf\" (UID: \"904d622c-9c62-4b50-8d74-df32398e7f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" Nov 03 01:27:44.285185 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.285153 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" Nov 03 01:27:44.653007 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:44.652979 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf"] Nov 03 01:27:44.655611 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:27:44.655584 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904d622c_9c62_4b50_8d74_df32398e7f4a.slice/crio-ec51dc752bf5ddcc4dc24b246da5196206f39a9d71858a939d9d685821e5811a WatchSource:0}: Error finding container ec51dc752bf5ddcc4dc24b246da5196206f39a9d71858a939d9d685821e5811a: Status 404 returned error can't find the container with id ec51dc752bf5ddcc4dc24b246da5196206f39a9d71858a939d9d685821e5811a Nov 03 01:27:45.653775 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:45.653728 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" event={"ID":"904d622c-9c62-4b50-8d74-df32398e7f4a","Type":"ContainerStarted","Data":"35a207c79983aba27f4d88466f3c2d1bd5da1804a226c9017aef5cdeaeea7e00"} Nov 03 01:27:45.654505 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:45.654471 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" event={"ID":"904d622c-9c62-4b50-8d74-df32398e7f4a","Type":"ContainerStarted","Data":"ec51dc752bf5ddcc4dc24b246da5196206f39a9d71858a939d9d685821e5811a"} Nov 03 01:27:45.769480 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:45.769134 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" podStartSLOduration=2.769113647 podStartE2EDuration="2.769113647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:45.768147343 +0000 UTC m=+4912.719087505" watchObservedRunningTime="2026-11-03 01:27:45.769113647 +0000 UTC m=+4912.720053807" Nov 03 01:27:46.660630 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:46.660595 6400 generic.go:356] "Generic (PLEG): container finished" podID="904d622c-9c62-4b50-8d74-df32398e7f4a" containerID="35a207c79983aba27f4d88466f3c2d1bd5da1804a226c9017aef5cdeaeea7e00" exitCode=0 Nov 03 01:27:46.661092 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:46.660668 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" event={"ID":"904d622c-9c62-4b50-8d74-df32398e7f4a","Type":"ContainerDied","Data":"35a207c79983aba27f4d88466f3c2d1bd5da1804a226c9017aef5cdeaeea7e00"} Nov 03 01:27:47.804228 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:47.804201 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" Nov 03 01:27:47.957521 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:47.957435 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-openshift-ca\") pod \"904d622c-9c62-4b50-8d74-df32398e7f4a\" (UID: \"904d622c-9c62-4b50-8d74-df32398e7f4a\") " Nov 03 01:27:47.957521 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:47.957477 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-xks-ca\") pod \"904d622c-9c62-4b50-8d74-df32398e7f4a\" (UID: \"904d622c-9c62-4b50-8d74-df32398e7f4a\") " Nov 03 01:27:47.957755 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:47.957605 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904d622c-9c62-4b50-8d74-df32398e7f4a-kube-api-access-rq4md\" (UniqueName: \"kubernetes.io/projected/904d622c-9c62-4b50-8d74-df32398e7f4a-kube-api-access-rq4md\") pod \"904d622c-9c62-4b50-8d74-df32398e7f4a\" (UID: \"904d622c-9c62-4b50-8d74-df32398e7f4a\") " Nov 03 01:27:47.957815 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:47.957796 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-openshift-ca" pod "904d622c-9c62-4b50-8d74-df32398e7f4a" (UID: "904d622c-9c62-4b50-8d74-df32398e7f4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:27:47.957968 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:47.957949 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-openshift-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:27:47.959695 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:47.959677 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904d622c-9c62-4b50-8d74-df32398e7f4a-kube-api-access-rq4md" pod "904d622c-9c62-4b50-8d74-df32398e7f4a" (UID: "904d622c-9c62-4b50-8d74-df32398e7f4a"). InnerVolumeSpecName "kube-api-access-rq4md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:27:47.959769 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:47.959753 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-xks-ca" pod "904d622c-9c62-4b50-8d74-df32398e7f4a" (UID: "904d622c-9c62-4b50-8d74-df32398e7f4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:27:48.058944 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:48.058915 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904d622c-9c62-4b50-8d74-df32398e7f4a-maas-api-xks-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:27:48.058944 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:48.058938 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq4md\" (UniqueName: \"kubernetes.io/projected/904d622c-9c62-4b50-8d74-df32398e7f4a-kube-api-access-rq4md\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:27:48.671984 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:48.671957 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" Nov 03 01:27:48.671984 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:48.671972 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf" event={"ID":"904d622c-9c62-4b50-8d74-df32398e7f4a","Type":"ContainerDied","Data":"ec51dc752bf5ddcc4dc24b246da5196206f39a9d71858a939d9d685821e5811a"} Nov 03 01:27:48.672205 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:48.672021 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec51dc752bf5ddcc4dc24b246da5196206f39a9d71858a939d9d685821e5811a" Nov 03 01:27:51.139673 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:51.139637 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:27:57.176942 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:27:57.176870 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:27:58.627605 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:58.627568 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:58.791375 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:27:58.791341 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:01.248343 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:01.248301 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:01.418389 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:01.418332 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:01.644424 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:01.644377 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:04.488330 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:04.488284 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:04.668111 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:04.668056 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:07.525401 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:07.525371 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:07.683030 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:07.682995 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:07.841510 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:07.841477 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:12.458517 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:12.458474 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:15.419190 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:15.419153 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:28:17.395550 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:17.395515 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:22.182211 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:22.182166 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:25.942437 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:25.942399 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:26.534939 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:26.534902 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:26.696738 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:26.696711 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:26.858898 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:26.858789 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:27.151410 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:28:27.151323 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:28:27.368252 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:27.368221 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:29.413295 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:29.413253 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:32.158599 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:32.158517 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:33.939897 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:33.939859 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:28:37.050836 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:37.050784 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:40.840126 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:40.840085 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:41.028394 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:41.028364 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:41.857014 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:41.856979 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:46.667721 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:46.667691 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:51.451495 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:51.451451 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:56.261792 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:28:56.261760 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:57.152085 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:28:57.152050 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:29:01.069411 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:01.069368 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:01.853296 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:01.853263 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:29:05.953058 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:05.953021 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:10.760359 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:10.760315 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:15.593915 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:15.593879 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:17.677531 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:17.677495 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:20.347067 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:20.347036 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:25.174809 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:25.174771 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:25.927227 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:25.927192 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:26.410941 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:26.410907 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:27.106726 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:27.106688 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:27.152377 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:29:27.152343 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:29:27.813407 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:27.813372 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:29:29.946458 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:29.946424 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:30.961627 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:30.961596 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:32.185159 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:32.185131 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:34.303936 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:34.303899 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:37.656125 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:37.656093 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:29:37.916781 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:37.916665 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:38.075131 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:29:38.075094 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:57.151594 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:29:57.151555 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:30:07.440474 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:30:07.440366 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:08.008861 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:30:08.008804 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:08.172070 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:30:08.172031 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:09.734154 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:30:09.734117 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:10.028772 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:30:10.028677 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:10.634777 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:30:10.634738 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:13.442715 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:30:13.442675 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:13.615068 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:30:13.615034 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:27.151505 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:30:27.151471 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:30:27.376081 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:30:27.376046 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:27.532367 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:30:27.532283 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:31.815250 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:30:31.815216 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:30:55.238967 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:30:55.238930 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:57.151039 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:30:57.150999 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:30:59.810960 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:30:59.810931 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:31:25.616085 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:31:25.616048 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:31:27.203455 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:31:27.203385 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:31:27.908276 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:31:27.908226 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:37.058073 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:31:37.058035 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:37.731280 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:31:37.731242 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:31:45.553493 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:31:45.553435 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:51.440640 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:31:51.440600 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:31:57.152185 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:31:57.152149 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:32:02.640668 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:32:02.640629 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:32:12.123193 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:32:12.123153 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:32:21.858562 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:32:21.858525 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:26.803006 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:32:26.802969 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:32:27.151901 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:32:27.151798 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:32:28.202536 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:32:28.202504 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:29.042546 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:32:29.042447 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f"] Nov 03 01:32:29.091955 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:32:29.091914 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478f117a-96k6f"] Nov 03 01:32:29.177543 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:32:29.177508 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7082571b-9ab9-4a36-a0c5-e2a5a9510917" path="/var/lib/kubelet/pods/7082571b-9ab9-4a36-a0c5-e2a5a9510917/volumes" Nov 03 01:32:48.051163 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:32:48.051130 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf"] Nov 03 01:32:48.100070 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:32:48.100040 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd26c0ac-2pthf"] Nov 03 01:32:49.177713 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:32:49.177679 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904d622c-9c62-4b50-8d74-df32398e7f4a" path="/var/lib/kubelet/pods/904d622c-9c62-4b50-8d74-df32398e7f4a/volumes" Nov 03 01:32:57.152807 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:32:57.152764 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:33:14.613938 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:33:14.613857 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:15.102021 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:33:15.101980 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:17.779441 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:33:17.779405 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:19.454157 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:33:19.454125 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:33:27.151696 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:33:27.151659 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:33:28.561863 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:33:28.561814 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:54.506165 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:33:54.506130 6400 scope.go:122] "RemoveContainer" containerID="35a207c79983aba27f4d88466f3c2d1bd5da1804a226c9017aef5cdeaeea7e00" Nov 03 01:33:54.514801 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:33:54.514779 6400 scope.go:122] "RemoveContainer" containerID="3c67174fc7979e357d295bf4cc8dc599f701eda99f00d9fbcfedcfff1bd4df6a" Nov 03 01:33:57.151209 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:33:57.151171 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:34:27.151677 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:34:27.151639 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:34:28.894133 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:34:28.894095 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:57.152337 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:34:57.152300 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:35:08.086213 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:35:08.086159 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:09.367268 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:35:09.367220 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:11.110109 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:35:11.110077 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:35:21.722584 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:35:21.722548 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:35:27.151527 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:35:27.151489 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:35:29.237075 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:35:29.237044 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:40.018119 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:35:40.018081 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:35:50.743699 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:35:50.743662 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:35:57.152211 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:35:57.152169 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:36:04.743297 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:36:04.743214 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:36:21.617649 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:36:21.617613 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:36:27.152029 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:36:27.151988 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:36:29.648085 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:36:29.648052 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:49.404353 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:36:49.404322 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:36:57.152228 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:36:57.152192 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:36:59.948349 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:36:59.948319 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:37:12.454864 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:37:12.454809 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:37:27.152583 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:37:27.152541 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:37:29.941830 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:37:29.941782 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:31.001221 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:37:31.001183 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:34.812496 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:37:34.812457 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:37:44.245251 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:37:44.245216 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:37:52.748809 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:37:52.748749 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:37:57.151905 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:37:57.151866 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:38:05.740786 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:38:05.740754 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:38:13.722362 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:38:13.722327 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:38:23.753365 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:38:23.753332 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:38:27.151490 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:38:27.151447 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:38:30.277891 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:38:30.277851 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:31.352873 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:38:31.352832 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:34.356594 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:38:34.356551 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:38:51.852081 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:38:51.852047 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:38:57.150615 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:38:57.150576 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:39:02.747022 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:39:02.746946 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:39:15.942902 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:39:15.942865 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:39:26.134686 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:39:26.134639 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:39:27.152172 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:39:27.152135 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:39:30.631038 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:39:30.631005 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:31.679728 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:39:31.679692 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:35.354990 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:39:35.354946 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:39:35.355495 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:39:35.355005 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs" podUID="ceaafa9a-74d0-40cf-8324-7c4f3cdd663f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:36.036779 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:39:36.036739 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:39:56.132730 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:39:56.132697 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:39:57.158296 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:39:57.158232 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:40:06.840321 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:06.840286 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:40:25.533197 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:25.533164 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:40:26.869428 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:26.869394 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:27.181685 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:40:27.181596 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:40:29.517252 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:29.517215 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:30.968480 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:30.968438 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:31.914286 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:31.914256 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:32.125608 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:32.125576 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:35.165599 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:35.165562 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:35.343059 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:35.343006 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:38.308994 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:38.308951 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:38.545196 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:38.545162 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:38.696181 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:38.696098 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:39.110143 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:39.110109 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:40:43.258494 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:43.258458 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:48.155355 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:48.155310 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:50.010716 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:50.010684 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:40:52.956436 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:52.956400 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:56.667652 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:56.667615 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:57.151840 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:40:57.151787 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:40:57.188706 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:57.188667 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:57.364681 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:57.364647 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:58.119199 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:40:58.119164 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:00.023650 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:00.023617 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:02.962788 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:02.962756 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:04.523719 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:04.523677 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:41:07.841341 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:07.841304 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:12.667212 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:12.667172 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:14.298696 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:14.298660 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:17.575272 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:17.575232 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:22.352063 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:22.352028 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:27.151970 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:41:27.151930 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:41:27.237887 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:27.237750 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:31.392900 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:31.392865 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:32.065621 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:32.065584 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:32.365970 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:32.365885 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:32.552088 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:32.552055 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:41:36.997789 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:36.997749 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:41.805268 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:41.805230 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:46.665939 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:46.665897 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:51.451812 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:51.451775 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:56.368865 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:56.368808 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:57.153036 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:41:57.152997 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:41:58.836742 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:58.836698 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:59.344329 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:41:59.344291 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:01.153216 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:01.153180 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:01.941587 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:01.941549 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:42:03.866656 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:03.866621 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:05.031101 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:05.031068 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:05.597650 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:05.597620 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:06.145030 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:06.144991 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:06.353502 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:06.353468 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:10.564408 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:10.564362 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:10.776699 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:10.776665 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:15.853399 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:15.853366 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:42:27.151079 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:42:27.151046 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:42:31.655612 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:31.655574 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:32.710036 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:32.709991 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:40.491317 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:40.491275 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:40.670124 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:40.670093 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:40.904952 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:40.904918 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:41.480238 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:41.480197 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:42.009637 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:42.009602 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:42.419169 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:42.419128 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:43.028630 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:43.028597 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:47.111244 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:47.111207 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:47.305595 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:47.305560 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:57.150994 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:42:57.150955 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:42:59.439554 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:42:59.439520 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:43:00.003845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:43:00.000659 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:00.161411 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:43:00.161376 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:27.151308 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:43:27.151269 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:43:32.018559 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:43:32.018470 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:33.113421 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:43:33.113377 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:57.152464 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:43:57.152421 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:44:27.151961 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:44:27.151926 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:44:32.347982 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:44:32.347943 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:33.407001 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:44:33.406964 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:33.515272 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:44:33.515244 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:44:46.346031 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:44:46.345991 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:46.516777 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:44:46.516736 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:48.010630 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:44:48.010592 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:44:49.929847 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:44:49.929019 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:54.587845 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:44:54.582604 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:57.151081 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:44:57.151042 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:44:57.639359 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:44:57.639327 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:45:06.135236 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:06.135154 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:45:09.624968 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.624935 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2"] Nov 03 01:45:09.625355 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.625346 6400 cpu_manager.go:414] "containerMap: removing container" podUID="904d622c-9c62-4b50-8d74-df32398e7f4a" containerName="revoke-keys" Nov 03 01:45:09.625398 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.625357 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904d622c-9c62-4b50-8d74-df32398e7f4a" containerName="revoke-keys" Nov 03 01:45:09.625446 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.625434 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="904d622c-9c62-4b50-8d74-df32398e7f4a" containerName="revoke-keys" Nov 03 01:45:09.628590 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.628572 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:09.678484 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.678452 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-maas-api-tls\") pod \"maas-api-d8998cc8c-m2dr2\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:09.678645 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.678626 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d578a51-e7bd-49b0-8207-a792a8ea9569-openshift-service-ca\") pod \"maas-api-d8998cc8c-m2dr2\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:09.678714 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.678662 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-metrics-certs\") pod \"maas-api-d8998cc8c-m2dr2\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:09.678714 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.678681 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdsww\" (UniqueName: \"kubernetes.io/projected/0d578a51-e7bd-49b0-8207-a792a8ea9569-kube-api-access-wdsww\") pod \"maas-api-d8998cc8c-m2dr2\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:09.732916 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.732884 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kgpd5\"" Nov 03 01:45:09.733105 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.732983 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 01:45:09.733105 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.732990 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 01:45:09.733105 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.733031 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:45:09.733105 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.733061 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:45:09.734194 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.734175 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2"] Nov 03 01:45:09.779665 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.779637 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d578a51-e7bd-49b0-8207-a792a8ea9569-openshift-service-ca\") pod \"maas-api-d8998cc8c-m2dr2\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:09.779853 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.779670 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-metrics-certs\") pod \"maas-api-d8998cc8c-m2dr2\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:09.779853 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.779687 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdsww\" (UniqueName: \"kubernetes.io/projected/0d578a51-e7bd-49b0-8207-a792a8ea9569-kube-api-access-wdsww\") pod \"maas-api-d8998cc8c-m2dr2\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:09.779853 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.779710 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-maas-api-tls\") pod \"maas-api-d8998cc8c-m2dr2\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:09.780368 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.780345 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d578a51-e7bd-49b0-8207-a792a8ea9569-openshift-service-ca\") pod \"maas-api-d8998cc8c-m2dr2\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:09.782302 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.782280 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-maas-api-tls\") pod \"maas-api-d8998cc8c-m2dr2\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:09.782429 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.782413 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-metrics-certs\") pod \"maas-api-d8998cc8c-m2dr2\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:09.836586 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.836563 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdsww\" (UniqueName: \"kubernetes.io/projected/0d578a51-e7bd-49b0-8207-a792a8ea9569-kube-api-access-wdsww\") pod \"maas-api-d8998cc8c-m2dr2\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:09.940793 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:09.940714 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:10.308650 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:10.308624 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2"] Nov 03 01:45:10.311469 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:45:10.311447 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d578a51_e7bd_49b0_8207_a792a8ea9569.slice/crio-90bf595a11ea66a979642632c2bcb2f9c9ca2bcfd97d156df0b03e88e527c824 WatchSource:0}: Error finding container 90bf595a11ea66a979642632c2bcb2f9c9ca2bcfd97d156df0b03e88e527c824: Status 404 returned error can't find the container with id 90bf595a11ea66a979642632c2bcb2f9c9ca2bcfd97d156df0b03e88e527c824 Nov 03 01:45:10.312917 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:10.312900 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:45:10.825835 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:10.825788 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" event={"ID":"0d578a51-e7bd-49b0-8207-a792a8ea9569","Type":"ContainerStarted","Data":"90bf595a11ea66a979642632c2bcb2f9c9ca2bcfd97d156df0b03e88e527c824"} Nov 03 01:45:11.831722 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:11.831691 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" event={"ID":"0d578a51-e7bd-49b0-8207-a792a8ea9569","Type":"ContainerStarted","Data":"0536020381a02c58d96110b0c27d1e1cc53994c68797fc81e155e86100c29a69"} Nov 03 01:45:11.832242 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:11.831772 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:11.945895 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:11.945846 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" podStartSLOduration=2.080099686 podStartE2EDuration="2.945812114s" totalImagesPullingTime="865.712428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:11.945296758 +0000 UTC m=+5958.896236917" watchObservedRunningTime="2026-11-03 01:45:11.945812114 +0000 UTC m=+5958.896752272" Nov 03 01:45:17.841793 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:17.841764 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:18.201736 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.201653 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4"] Nov 03 01:45:18.205341 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.205320 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.261344 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.261310 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c36f163-833d-4e14-a25a-653e5689ed4f-openshift-service-ca\") pod \"maas-api-74648cd7d7-lnqs4\" (UID: \"7c36f163-833d-4e14-a25a-653e5689ed4f\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.261563 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.261542 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c36f163-833d-4e14-a25a-653e5689ed4f-maas-api-tls\") pod \"maas-api-74648cd7d7-lnqs4\" (UID: \"7c36f163-833d-4e14-a25a-653e5689ed4f\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.261728 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.261704 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c36f163-833d-4e14-a25a-653e5689ed4f-metrics-certs\") pod \"maas-api-74648cd7d7-lnqs4\" (UID: \"7c36f163-833d-4e14-a25a-653e5689ed4f\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.261813 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.261766 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4vhl\" (UniqueName: \"kubernetes.io/projected/7c36f163-833d-4e14-a25a-653e5689ed4f-kube-api-access-s4vhl\") pod \"maas-api-74648cd7d7-lnqs4\" (UID: \"7c36f163-833d-4e14-a25a-653e5689ed4f\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.261905 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.261876 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4"] Nov 03 01:45:18.363058 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.363022 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c36f163-833d-4e14-a25a-653e5689ed4f-openshift-service-ca\") pod \"maas-api-74648cd7d7-lnqs4\" (UID: \"7c36f163-833d-4e14-a25a-653e5689ed4f\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.363265 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.363068 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c36f163-833d-4e14-a25a-653e5689ed4f-maas-api-tls\") pod \"maas-api-74648cd7d7-lnqs4\" (UID: \"7c36f163-833d-4e14-a25a-653e5689ed4f\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.363265 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.363124 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c36f163-833d-4e14-a25a-653e5689ed4f-metrics-certs\") pod \"maas-api-74648cd7d7-lnqs4\" (UID: \"7c36f163-833d-4e14-a25a-653e5689ed4f\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.363265 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.363141 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4vhl\" (UniqueName: \"kubernetes.io/projected/7c36f163-833d-4e14-a25a-653e5689ed4f-kube-api-access-s4vhl\") pod \"maas-api-74648cd7d7-lnqs4\" (UID: \"7c36f163-833d-4e14-a25a-653e5689ed4f\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.363725 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.363703 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c36f163-833d-4e14-a25a-653e5689ed4f-openshift-service-ca\") pod \"maas-api-74648cd7d7-lnqs4\" (UID: \"7c36f163-833d-4e14-a25a-653e5689ed4f\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.365737 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.365714 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c36f163-833d-4e14-a25a-653e5689ed4f-metrics-certs\") pod \"maas-api-74648cd7d7-lnqs4\" (UID: \"7c36f163-833d-4e14-a25a-653e5689ed4f\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.365846 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.365754 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c36f163-833d-4e14-a25a-653e5689ed4f-maas-api-tls\") pod \"maas-api-74648cd7d7-lnqs4\" (UID: \"7c36f163-833d-4e14-a25a-653e5689ed4f\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.419697 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.419666 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4vhl\" (UniqueName: \"kubernetes.io/projected/7c36f163-833d-4e14-a25a-653e5689ed4f-kube-api-access-s4vhl\") pod \"maas-api-74648cd7d7-lnqs4\" (UID: \"7c36f163-833d-4e14-a25a-653e5689ed4f\") " pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.517970 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.517876 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:18.885634 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:18.885608 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4"] Nov 03 01:45:18.887857 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:45:18.887831 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c36f163_833d_4e14_a25a_653e5689ed4f.slice/crio-ed464051cfe1f3ee06c92d6d8032eda56935e68911e5b06ee9873c00123a283e WatchSource:0}: Error finding container ed464051cfe1f3ee06c92d6d8032eda56935e68911e5b06ee9873c00123a283e: Status 404 returned error can't find the container with id ed464051cfe1f3ee06c92d6d8032eda56935e68911e5b06ee9873c00123a283e Nov 03 01:45:19.869028 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:19.868983 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" event={"ID":"7c36f163-833d-4e14-a25a-653e5689ed4f","Type":"ContainerStarted","Data":"6869c79caf6806c44887926a6af74cce4c0e6d047ab402c765e4449d810579f8"} Nov 03 01:45:19.869166 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:19.869054 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" event={"ID":"7c36f163-833d-4e14-a25a-653e5689ed4f","Type":"ContainerStarted","Data":"ed464051cfe1f3ee06c92d6d8032eda56935e68911e5b06ee9873c00123a283e"} Nov 03 01:45:19.869166 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:19.869099 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:20.001588 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:20.001516 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" podStartSLOduration=1.110197729 podStartE2EDuration="2.001500785s" totalImagesPullingTime="891.303056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:20.001098494 +0000 UTC m=+5966.952038655" watchObservedRunningTime="2026-11-03 01:45:20.001500785 +0000 UTC m=+5966.952440934" Nov 03 01:45:22.250139 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:22.250103 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:45:25.879685 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:25.879653 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74648cd7d7-lnqs4" Nov 03 01:45:26.044038 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.044002 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2"] Nov 03 01:45:26.044239 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.044220 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" podUID="0d578a51-e7bd-49b0-8207-a792a8ea9569" containerName="maas-api" containerID="cri-o://0536020381a02c58d96110b0c27d1e1cc53994c68797fc81e155e86100c29a69" gracePeriod=30 Nov 03 01:45:26.295725 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.295700 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:26.463466 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.463385 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-metrics-certs\") pod \"0d578a51-e7bd-49b0-8207-a792a8ea9569\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " Nov 03 01:45:26.463466 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.463451 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d578a51-e7bd-49b0-8207-a792a8ea9569-kube-api-access-wdsww\" (UniqueName: \"kubernetes.io/projected/0d578a51-e7bd-49b0-8207-a792a8ea9569-kube-api-access-wdsww\") pod \"0d578a51-e7bd-49b0-8207-a792a8ea9569\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " Nov 03 01:45:26.463728 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.463484 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-maas-api-tls\") pod \"0d578a51-e7bd-49b0-8207-a792a8ea9569\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " Nov 03 01:45:26.463728 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.463518 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d578a51-e7bd-49b0-8207-a792a8ea9569-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d578a51-e7bd-49b0-8207-a792a8ea9569-openshift-service-ca\") pod \"0d578a51-e7bd-49b0-8207-a792a8ea9569\" (UID: \"0d578a51-e7bd-49b0-8207-a792a8ea9569\") " Nov 03 01:45:26.463968 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.463945 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d578a51-e7bd-49b0-8207-a792a8ea9569-openshift-service-ca" pod "0d578a51-e7bd-49b0-8207-a792a8ea9569" (UID: "0d578a51-e7bd-49b0-8207-a792a8ea9569"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:26.465649 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.465619 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-metrics-certs" pod "0d578a51-e7bd-49b0-8207-a792a8ea9569" (UID: "0d578a51-e7bd-49b0-8207-a792a8ea9569"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:26.465743 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.465690 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-maas-api-tls" pod "0d578a51-e7bd-49b0-8207-a792a8ea9569" (UID: "0d578a51-e7bd-49b0-8207-a792a8ea9569"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:26.465783 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.465754 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d578a51-e7bd-49b0-8207-a792a8ea9569-kube-api-access-wdsww" pod "0d578a51-e7bd-49b0-8207-a792a8ea9569" (UID: "0d578a51-e7bd-49b0-8207-a792a8ea9569"). InnerVolumeSpecName "kube-api-access-wdsww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:26.565095 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.565054 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d578a51-e7bd-49b0-8207-a792a8ea9569-openshift-service-ca\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:26.565095 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.565090 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-metrics-certs\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:26.565095 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.565101 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdsww\" (UniqueName: \"kubernetes.io/projected/0d578a51-e7bd-49b0-8207-a792a8ea9569-kube-api-access-wdsww\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:26.565435 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.565110 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d578a51-e7bd-49b0-8207-a792a8ea9569-maas-api-tls\") on node \"ip-10-0-135-199.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:26.924243 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.924212 6400 generic.go:356] "Generic (PLEG): container finished" podID="0d578a51-e7bd-49b0-8207-a792a8ea9569" containerID="0536020381a02c58d96110b0c27d1e1cc53994c68797fc81e155e86100c29a69" exitCode=0 Nov 03 01:45:26.924706 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.924255 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" Nov 03 01:45:26.924706 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.924293 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" event={"ID":"0d578a51-e7bd-49b0-8207-a792a8ea9569","Type":"ContainerDied","Data":"0536020381a02c58d96110b0c27d1e1cc53994c68797fc81e155e86100c29a69"} Nov 03 01:45:26.924706 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.924347 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2" event={"ID":"0d578a51-e7bd-49b0-8207-a792a8ea9569","Type":"ContainerDied","Data":"90bf595a11ea66a979642632c2bcb2f9c9ca2bcfd97d156df0b03e88e527c824"} Nov 03 01:45:26.924706 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.924371 6400 scope.go:122] "RemoveContainer" containerID="0536020381a02c58d96110b0c27d1e1cc53994c68797fc81e155e86100c29a69" Nov 03 01:45:26.933810 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.933794 6400 scope.go:122] "RemoveContainer" containerID="0536020381a02c58d96110b0c27d1e1cc53994c68797fc81e155e86100c29a69" Nov 03 01:45:26.934164 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:45:26.934134 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0536020381a02c58d96110b0c27d1e1cc53994c68797fc81e155e86100c29a69\": container with ID starting with 0536020381a02c58d96110b0c27d1e1cc53994c68797fc81e155e86100c29a69 not found: ID does not exist" containerID="0536020381a02c58d96110b0c27d1e1cc53994c68797fc81e155e86100c29a69" Nov 03 01:45:26.934262 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:26.934173 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0536020381a02c58d96110b0c27d1e1cc53994c68797fc81e155e86100c29a69"} err="failed to get container status \"0536020381a02c58d96110b0c27d1e1cc53994c68797fc81e155e86100c29a69\": rpc error: code = NotFound desc = could not find container \"0536020381a02c58d96110b0c27d1e1cc53994c68797fc81e155e86100c29a69\": container with ID starting with 0536020381a02c58d96110b0c27d1e1cc53994c68797fc81e155e86100c29a69 not found: ID does not exist" Nov 03 01:45:27.092515 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:27.092483 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2"] Nov 03 01:45:27.098933 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:27.098906 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8998cc8c-m2dr2"] Nov 03 01:45:27.152314 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:45:27.152280 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:45:27.177302 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:27.177240 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d578a51-e7bd-49b0-8207-a792a8ea9569" path="/var/lib/kubelet/pods/0d578a51-e7bd-49b0-8207-a792a8ea9569/volumes" Nov 03 01:45:32.733147 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:32.733112 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:34.399773 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:34.399726 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:57.152366 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:45:57.152325 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:45:57.543567 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:45:57.543531 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:46:09.750875 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:46:09.750839 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:46:27.152543 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:46:27.152506 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:46:33.102082 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:46:33.102008 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:34.085899 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:46:34.085858 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:34.810449 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:46:34.810414 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:46:46.932082 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:46:46.932045 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:46:57.156159 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:46:57.156122 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:47:17.952201 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:47:17.952171 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:47:27.152107 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:47:27.152073 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:47:28.633034 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:47:28.632998 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:47:33.391124 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:47:33.391085 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:34.415408 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:47:34.415356 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:41.130015 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:47:41.129983 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:47:53.133761 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:47:53.133722 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:47:57.150975 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:47:57.150936 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:48:14.597235 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:48:14.597138 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:14.809775 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:48:14.809737 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:16.648490 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:48:16.648454 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:48:27.151440 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:48:27.151400 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:48:27.311683 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:48:27.311647 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:48:33.838848 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:48:33.837954 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:34.849861 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:48:34.849800 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:37.037540 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:48:37.037505 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:48:56.447300 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:48:56.447264 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:48:57.160891 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:48:57.160812 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:49:20.327647 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:49:20.327614 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:49:27.152322 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:49:27.152285 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:49:33.840959 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:49:33.840862 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:34.175987 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:49:34.175891 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:35.259538 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:49:35.259500 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:57.152588 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:49:57.152549 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:50:27.150758 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:50:27.150723 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:50:34.548118 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:50:34.548071 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:36.296687 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:50:36.296651 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:48.881045 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:50:48.880995 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:50:48.881527 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:50:48.881116 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk" podUID="956c60bc-c036-450b-85d9-ffd5f5a0971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:50:57.167608 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:50:57.167566 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:51:18.240332 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:51:18.240249 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8d9rl"] Nov 03 01:51:19.137847 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:51:19.134433 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:19.302534 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:51:19.302501 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:20.062246 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:51:20.062211 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:27.151393 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:51:27.151358 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:51:34.928632 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:51:34.928593 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:35.890699 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:51:35.890659 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:50.071684 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:51:50.071648 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-62pxm_288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1/manager/0.log" Nov 03 01:51:50.818223 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:51:50.818190 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bg4db_9adacad7-56f7-41dc-90c2-0dbdab888504/manager/0.log" Nov 03 01:51:53.502301 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:51:53.502267 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-74648cd7d7-lnqs4_7c36f163-833d-4e14-a25a-653e5689ed4f/maas-api/0.log" Nov 03 01:51:57.151983 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:51:57.151947 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:52:01.196439 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:01.196408 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-8d9rl_fc80ce75-cd2c-43c0-888b-4e41bbd9e047/limitador/0.log" Nov 03 01:52:02.685215 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:02.685174 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bcbbp_526962f1-ecc2-44db-9485-d25e2082ec97/discovery/0.log" Nov 03 01:52:03.980576 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:03.980544 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs_ceaafa9a-74d0-40cf-8324-7c4f3cdd663f/istio-proxy/0.log" Nov 03 01:52:04.857619 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:04.857584 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk_956c60bc-c036-450b-85d9-ffd5f5a0971e/istio-proxy/0.log" Nov 03 01:52:05.975752 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:05.975715 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-xsttm_b2266845-4cd0-4d7e-b244-1f6bfd536847/istio-proxy/0.log" Nov 03 01:52:21.138925 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:21.138889 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:21.340889 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:21.340847 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:27.151458 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:52:27.151419 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:52:29.902715 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:29.902681 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-52cjz_fdf82aaa-e340-4a97-a3f4-297694eccab3/global-pull-secret-syncer/0.log" Nov 03 01:52:32.914081 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:32.914001 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-62pxm_288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1/manager/0.log" Nov 03 01:52:32.985272 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:32.985237 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" Nov 03 01:52:34.126099 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:34.126056 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bg4db_9adacad7-56f7-41dc-90c2-0dbdab888504/manager/0.log" Nov 03 01:52:34.379259 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:34.379156 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-199.us-west-1.compute.internal_cf58af1ed7bc599c71fbbaa4e8065ef6/haproxy/0.log" Nov 03 01:52:35.292775 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:35.292732 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:36.231992 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:36.231959 6400 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-njmnx_7314649e-0fd8-4bcd-bcfa-5140fb2711cf/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:38.753247 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:38.753208 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-62pxm_288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1/manager/0.log" Nov 03 01:52:39.790616 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.790573 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26269/perf-node-gather-daemonset-5txh6"] Nov 03 01:52:39.791211 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.791191 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0d578a51-e7bd-49b0-8207-a792a8ea9569" containerName="maas-api" Nov 03 01:52:39.791301 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.791214 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d578a51-e7bd-49b0-8207-a792a8ea9569" containerName="maas-api" Nov 03 01:52:39.791359 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.791310 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d578a51-e7bd-49b0-8207-a792a8ea9569" containerName="maas-api" Nov 03 01:52:39.794537 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.794511 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.897378 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.897339 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26269\"/\"openshift-service-ca.crt\"" Nov 03 01:52:39.897378 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.897339 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26269\"/\"kube-root-ca.crt\"" Nov 03 01:52:39.897673 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.897532 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/240c84a7-61b7-4b88-bb66-2af82f425881-podres\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.897673 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.897548 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-26269\"/\"default-dockercfg-bj4wk\"" Nov 03 01:52:39.897673 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.897585 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/240c84a7-61b7-4b88-bb66-2af82f425881-sys\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.897904 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.897744 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjf67\" (UniqueName: \"kubernetes.io/projected/240c84a7-61b7-4b88-bb66-2af82f425881-kube-api-access-mjf67\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.897904 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.897807 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/240c84a7-61b7-4b88-bb66-2af82f425881-lib-modules\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.898033 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.897985 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/240c84a7-61b7-4b88-bb66-2af82f425881-proc\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.899193 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.899172 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26269/perf-node-gather-daemonset-5txh6"] Nov 03 01:52:39.998663 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.998622 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/240c84a7-61b7-4b88-bb66-2af82f425881-proc\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.998921 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.998722 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/240c84a7-61b7-4b88-bb66-2af82f425881-podres\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.998921 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.998751 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/240c84a7-61b7-4b88-bb66-2af82f425881-proc\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.998921 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.998766 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/240c84a7-61b7-4b88-bb66-2af82f425881-sys\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.998921 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.998895 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/240c84a7-61b7-4b88-bb66-2af82f425881-podres\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.998921 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.998834 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/240c84a7-61b7-4b88-bb66-2af82f425881-sys\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.999180 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.998922 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjf67\" (UniqueName: \"kubernetes.io/projected/240c84a7-61b7-4b88-bb66-2af82f425881-kube-api-access-mjf67\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.999180 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.998969 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/240c84a7-61b7-4b88-bb66-2af82f425881-lib-modules\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:39.999180 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:39.999126 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/240c84a7-61b7-4b88-bb66-2af82f425881-lib-modules\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:40.019040 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:40.019006 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bg4db_9adacad7-56f7-41dc-90c2-0dbdab888504/manager/0.log" Nov 03 01:52:40.057321 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:40.057233 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjf67\" (UniqueName: \"kubernetes.io/projected/240c84a7-61b7-4b88-bb66-2af82f425881-kube-api-access-mjf67\") pod \"perf-node-gather-daemonset-5txh6\" (UID: \"240c84a7-61b7-4b88-bb66-2af82f425881\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:40.110646 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:40.110609 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:40.480619 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:40.480592 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26269/perf-node-gather-daemonset-5txh6"] Nov 03 01:52:40.483114 ip-10-0-135-199 kubenswrapper[6400]: W1103 01:52:40.483067 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod240c84a7_61b7_4b88_bb66_2af82f425881.slice/crio-eed1d14a3134a8d04cff835fe71ad5ac8cd6eb2b695fe88face1693967eb206a WatchSource:0}: Error finding container eed1d14a3134a8d04cff835fe71ad5ac8cd6eb2b695fe88face1693967eb206a: Status 404 returned error can't find the container with id eed1d14a3134a8d04cff835fe71ad5ac8cd6eb2b695fe88face1693967eb206a Nov 03 01:52:41.299590 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:41.299549 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" event={"ID":"240c84a7-61b7-4b88-bb66-2af82f425881","Type":"ContainerStarted","Data":"72afd29d85d11f1831ad0f3fb3c25db7412c3c62a40eaa4745f3cfc4f12131c4"} Nov 03 01:52:41.300122 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:41.299603 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" event={"ID":"240c84a7-61b7-4b88-bb66-2af82f425881","Type":"ContainerStarted","Data":"eed1d14a3134a8d04cff835fe71ad5ac8cd6eb2b695fe88face1693967eb206a"} Nov 03 01:52:41.300122 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:41.299635 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:41.414695 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:41.414638 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" podStartSLOduration=2.414617034 podStartE2EDuration="2.414617034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:41.413646428 +0000 UTC m=+6408.364586584" watchObservedRunningTime="2026-11-03 01:52:41.414617034 +0000 UTC m=+6408.365557221" Nov 03 01:52:42.363629 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:42.363597 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-8d9rl_fc80ce75-cd2c-43c0-888b-4e41bbd9e047/limitador/0.log" Nov 03 01:52:44.560095 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:44.560054 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-62pxm_288ffc8d-dc5b-4bd0-a9ef-bb45712f56a1/manager/0.log" Nov 03 01:52:45.336417 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:45.336379 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_990360fa-7788-4af4-8d22-47b4452f6a7c/alertmanager/0.log" Nov 03 01:52:45.807410 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:45.807369 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_990360fa-7788-4af4-8d22-47b4452f6a7c/config-reloader/0.log" Nov 03 01:52:45.885873 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:45.885843 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bg4db_9adacad7-56f7-41dc-90c2-0dbdab888504/manager/0.log" Nov 03 01:52:46.220719 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:46.220632 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_990360fa-7788-4af4-8d22-47b4452f6a7c/kube-rbac-proxy-web/0.log" Nov 03 01:52:46.632536 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:46.632500 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_990360fa-7788-4af4-8d22-47b4452f6a7c/kube-rbac-proxy/0.log" Nov 03 01:52:47.045358 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:47.045333 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_990360fa-7788-4af4-8d22-47b4452f6a7c/kube-rbac-proxy-metric/0.log" Nov 03 01:52:47.315079 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:47.314999 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-26269/perf-node-gather-daemonset-5txh6" Nov 03 01:52:47.457178 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:47.457140 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_990360fa-7788-4af4-8d22-47b4452f6a7c/prom-label-proxy/0.log" Nov 03 01:52:47.868882 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:47.868851 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_990360fa-7788-4af4-8d22-47b4452f6a7c/init-config-reloader/0.log" Nov 03 01:52:48.708696 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:48.708665 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7p54s_8ba97f37-1485-4215-abe3-3c9083759d8a/kube-state-metrics/0.log" Nov 03 01:52:49.110083 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:49.110054 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7p54s_8ba97f37-1485-4215-abe3-3c9083759d8a/kube-rbac-proxy-main/0.log" Nov 03 01:52:49.521272 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:49.521199 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7p54s_8ba97f37-1485-4215-abe3-3c9083759d8a/kube-rbac-proxy-self/0.log" Nov 03 01:52:49.940556 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:49.940528 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-745647dff6-lq2qf_9ee339b2-b471-4efd-829e-dd247c78685a/metrics-server/0.log" Nov 03 01:52:50.355200 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:50.355165 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-vcbjm_443202b2-a46c-41af-b097-93658ee871f8/monitoring-plugin/0.log" Nov 03 01:52:52.029010 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:52.028952 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzcwb_9967e995-dc16-4869-bb2f-02eb340eba3e/node-exporter/0.log" Nov 03 01:52:52.446184 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:52.446156 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzcwb_9967e995-dc16-4869-bb2f-02eb340eba3e/kube-rbac-proxy/0.log" Nov 03 01:52:52.857374 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:52.857339 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzcwb_9967e995-dc16-4869-bb2f-02eb340eba3e/init-textfile/0.log" Nov 03 01:52:53.460809 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:53.460774 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bcbbp_526962f1-ecc2-44db-9485-d25e2082ec97/discovery/0.log" Nov 03 01:52:54.428962 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:54.428924 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs_ceaafa9a-74d0-40cf-8324-7c4f3cdd663f/istio-proxy/0.log" Nov 03 01:52:54.516806 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:54.516767 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2psfk_54b0b5df-4723-4f99-98a0-b7a386f3d0c5/kube-rbac-proxy-main/0.log" Nov 03 01:52:54.938264 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:54.938232 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2psfk_54b0b5df-4723-4f99-98a0-b7a386f3d0c5/kube-rbac-proxy-self/0.log" Nov 03 01:52:55.175539 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:55.175487 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-mmvpk_956c60bc-c036-450b-85d9-ffd5f5a0971e/istio-proxy/0.log" Nov 03 01:52:55.349713 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:55.349676 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2psfk_54b0b5df-4723-4f99-98a0-b7a386f3d0c5/openshift-state-metrics/0.log" Nov 03 01:52:56.027330 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:56.027299 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-xsttm_b2266845-4cd0-4d7e-b244-1f6bfd536847/istio-proxy/0.log" Nov 03 01:52:57.151287 ip-10-0-135-199 kubenswrapper[6400]: E1103 01:52:57.151253 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6v6bq" containerName="extractor" Nov 03 01:52:58.672225 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:58.672196 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-jm5n9_169c66f9-91af-4e46-bf67-019e0cec4a43/prometheus-operator/0.log" Nov 03 01:52:59.079785 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:59.079745 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-jm5n9_169c66f9-91af-4e46-bf67-019e0cec4a43/kube-rbac-proxy/0.log" Nov 03 01:52:59.914385 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:52:59.914356 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d6548ffc8-8tldc_d648f173-aa93-4fe7-999b-b50c88be3440/telemeter-client/0.log" Nov 03 01:53:00.325329 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:53:00.325299 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d6548ffc8-8tldc_d648f173-aa93-4fe7-999b-b50c88be3440/reload/0.log" Nov 03 01:53:00.739081 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:53:00.738992 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d6548ffc8-8tldc_d648f173-aa93-4fe7-999b-b50c88be3440/kube-rbac-proxy/0.log" Nov 03 01:53:01.159660 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:53:01.159632 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-869bd4cbfb-2qs6h_cbec544b-d079-48c5-9d21-0e739d7512d4/thanos-query/0.log" Nov 03 01:53:01.575878 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:53:01.575846 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-869bd4cbfb-2qs6h_cbec544b-d079-48c5-9d21-0e739d7512d4/kube-rbac-proxy-web/0.log" Nov 03 01:53:01.988872 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:53:01.988790 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-869bd4cbfb-2qs6h_cbec544b-d079-48c5-9d21-0e739d7512d4/kube-rbac-proxy/0.log" Nov 03 01:53:02.398674 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:53:02.398643 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-869bd4cbfb-2qs6h_cbec544b-d079-48c5-9d21-0e739d7512d4/prom-label-proxy/0.log" Nov 03 01:53:02.808775 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:53:02.808746 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-869bd4cbfb-2qs6h_cbec544b-d079-48c5-9d21-0e739d7512d4/kube-rbac-proxy-rules/0.log" Nov 03 01:53:02.945793 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:53:02.945760 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-8d9rl_fc80ce75-cd2c-43c0-888b-4e41bbd9e047/limitador/0.log" Nov 03 01:53:03.261476 ip-10-0-135-199 kubenswrapper[6400]: I1103 01:53:03.261357 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-869bd4cbfb-2qs6h_cbec544b-d079-48c5-9d21-0e739d7512d4/kube-rbac-proxy-metrics/0.log"